Nov 02 21:31:34.643215 ip-10-0-131-47 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 21:31:34.643224 ip-10-0-131-47 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 21:31:34.643232 ip-10-0-131-47 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 21:31:34.643483 ip-10-0-131-47 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 21:31:44.692597 ip-10-0-131-47 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 21:31:44.692611 ip-10-0-131-47 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 34d7e76190cb433c9d1e6431e3cb858c -- Nov 02 21:33:39.490069 ip-10-0-131-47 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:33:39.864734 ip-10-0-131-47 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:33:39.864734 ip-10-0-131-47 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:33:39.864734 ip-10-0-131-47 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:33:39.864734 ip-10-0-131-47 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870343 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870434 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870439 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870442 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870446 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870449 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.870440 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870452 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870456 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870459 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870462 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870465 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870468 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870470 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870473 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870477 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870483 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870487 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870490 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870494 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870502 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870505 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870508 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870511 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870513 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870516 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870519 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.870708 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870521 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870524 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870531 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870534 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870537 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870539 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870542 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870544 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870548 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870550 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870555 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870558 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870561 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870564 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870567 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870569 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870572 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870575 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.871293 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870577 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870579 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870582 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870586 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870590 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870594 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870597 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870600 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870603 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870606 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870608 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870611 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870613 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870616 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870619 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870622 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870625 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870627 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870631 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.871737 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870634 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870637 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870640 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870642 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870645 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870647 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870650 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870653 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870657 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870659 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870662 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870665 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870668 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870670 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870673 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870676 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870679 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870681 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870684 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.872225 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870693 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870695 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870698 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870701 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870703 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870706 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870708 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870711 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870714 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870717 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870719 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870722 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870724 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870727 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870729 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870732 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870734 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870738 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870758 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.872667 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870763 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870768 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870771 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.870773 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871632 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871639 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871642 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871645 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871647 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871650 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871653 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871656 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871659 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871662 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871670 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871673 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871676 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871678 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871681 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871684 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.873189 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871686 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871689 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871691 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871694 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871696 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871699 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871702 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871705 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871708 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871713 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871715 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871718 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871720 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871723 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871727 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871729 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871732 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871734 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871737 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.873685 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871755 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871758 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871761 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871763 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871767 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871769 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871772 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871781 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871784 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871786 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871789 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871791 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871794 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871796 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871799 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871801 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871804 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871807 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871810 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871813 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.874169 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871815 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871818 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871821 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871823 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871826 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871830 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871834 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871837 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871840 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871843 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871846 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871848 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871851 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871853 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871856 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871861 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871864 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871867 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.874634 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871869 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871879 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871882 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871885 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871888 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871890 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871893 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871895 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871898 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871901 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871906 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871909 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871913 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871916 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871919 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871921 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871924 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871926 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.875150 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871929 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871932 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871934 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871937 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871939 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871942 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871944 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871947 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871949 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871952 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871955 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.871957 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872494 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872501 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872504 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872507 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872510 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872512 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872515 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872518 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.875586 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872521 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872523 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872527 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872529 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872532 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872534 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872537 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872540 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872542 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872545 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872548 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872550 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872555 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872558 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872560 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872563 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872566 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872568 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872571 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.876080 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872573 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872576 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872581 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872585 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872588 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872592 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872595 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872601 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872604 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872606 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872609 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872611 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872614 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872616 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872619 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872622 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872624 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872627 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872629 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.876551 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872632 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872634 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872637 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872639 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872642 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872644 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872647 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872649 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872652 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872654 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872657 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872659 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872661 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872664 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872666 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872669 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872674 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872678 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872681 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.877037 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872684 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872687 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872691 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872694 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872696 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872698 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872701 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872703 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872706 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872709 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872712 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872714 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872717 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872719 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872722 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872725 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872727 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872730 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872732 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.877503 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872735 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872738 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872755 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872758 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872761 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872763 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872768 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872771 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872774 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872777 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872779 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872782 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872785 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872787 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872790 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872793 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872796 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872798 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.877971 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872801 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.872803 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872902 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872911 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872924 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872929 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872934 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872937 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872942 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872946 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872950 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872953 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872957 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872960 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872963 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872967 2580 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872970 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872973 2580 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872975 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872979 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872984 2580 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872987 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872990 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872993 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:33:39.878422 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.872997 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873001 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873004 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873007 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873011 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873014 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873017 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873020 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873024 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873027 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873033 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873036 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873039 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873042 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873045 2580 flags.go:64] FLAG: --enable-server="true" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873048 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873053 2580 flags.go:64] FLAG: --event-burst="100" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873056 2580 flags.go:64] FLAG: --event-qps="50" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873058 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873062 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873065 2580 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873069 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873072 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873075 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873078 2580 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:33:39.879007 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873081 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873084 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873087 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873091 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873095 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873098 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873101 2580 flags.go:64] FLAG: --feature-gates="" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873106 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873109 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873112 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873115 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873118 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873121 2580 flags.go:64] FLAG: --help="false" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873124 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873128 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873132 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873135 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873139 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873143 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873146 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873149 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873152 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873155 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:33:39.879658 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873158 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873161 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873164 2580 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873167 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873170 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873173 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873176 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873179 2580 flags.go:64] FLAG: --lock-file="" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873182 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873185 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873188 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873194 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873197 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873202 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873205 2580 flags.go:64] FLAG: --logging-format="text" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873208 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873211 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873214 2580 flags.go:64] FLAG: --manifest-url="" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873217 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873222 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873225 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873229 2580 flags.go:64] FLAG: --max-pods="110" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873232 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873235 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873239 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:33:39.880205 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873242 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873246 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873249 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873253 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873261 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873264 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873267 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873270 2580 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873273 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873276 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873279 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873282 2580 flags.go:64] FLAG: --port="10250" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873285 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873288 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-094e21b90004baa96" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873291 2580 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873294 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873298 2580 flags.go:64] FLAG: --register-node="true" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873301 2580 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873305 2580 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873308 2580 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873312 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873315 2580 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873319 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873323 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873326 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873329 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:33:39.880811 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873332 2580 flags.go:64] FLAG: --runonce="false" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873335 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873338 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873341 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873344 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873347 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873351 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873355 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873359 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873362 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873365 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873367 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873370 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873373 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873376 2580 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873379 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873384 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873387 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873390 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873395 2580 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873397 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873400 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873403 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873406 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873409 2580 flags.go:64] FLAG: --v="2" Nov 02 21:33:39.881409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873414 2580 flags.go:64] FLAG: --version="false" Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873419 2580 flags.go:64] FLAG: --vmodule="" Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873423 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873427 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873568 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873572 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873575 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873577 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873580 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873588 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873591 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873594 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873596 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873599 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873603 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873606 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873610 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873613 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873617 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.882001 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873619 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873622 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873625 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873627 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873630 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873633 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873636 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873638 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873641 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873644 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873646 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873649 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873651 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873654 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873658 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873660 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873663 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873666 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873668 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873671 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.882455 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873673 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873676 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873679 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873681 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873684 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873687 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873689 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873694 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873700 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873702 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873705 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873708 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873713 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873716 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873718 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873721 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873723 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873726 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.882957 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873728 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873731 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873734 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873736 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873751 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873755 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873757 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873761 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873764 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873767 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873769 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873772 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873775 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873777 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873780 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873782 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873785 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873788 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873790 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873793 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.883415 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873796 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873799 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873802 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873806 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873810 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873813 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873816 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873818 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873821 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873823 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873826 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873829 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873832 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873834 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873837 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873855 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873859 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873862 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873865 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873869 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.883956 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873873 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873875 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873878 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873881 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873884 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873887 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873890 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873893 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873895 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873898 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873900 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873903 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.873931 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873971 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873975 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873982 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.884420 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873985 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873988 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873990 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873993 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873996 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.873999 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874001 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874004 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874006 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874009 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874011 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874014 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874017 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874019 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874022 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874026 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874028 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874031 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874034 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874036 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.884818 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874039 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874041 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874044 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874046 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874049 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874052 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874056 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874059 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874062 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874066 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874069 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874072 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874075 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874077 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874080 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874083 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874085 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874088 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874090 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874093 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.885280 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874095 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874098 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874101 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874103 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874106 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874108 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874111 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874115 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874117 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874120 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874123 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874125 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874128 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874130 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874133 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874135 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874138 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874140 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874143 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874145 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.885754 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874147 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874151 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874154 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874156 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874159 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874163 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874165 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874170 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874174 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874177 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874180 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874182 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874185 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874188 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874191 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874193 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874196 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874199 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.886216 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874201 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874205 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874208 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874211 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874213 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874216 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874218 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874221 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874224 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874226 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874228 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874231 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874233 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874236 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874239 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874242 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874244 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874247 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874251 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874253 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.886639 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874256 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874259 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874261 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874264 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.874266 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.881112 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.881129 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.881142 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.881147 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881207 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881212 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881215 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881218 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881221 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881224 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881227 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881230 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881232 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881235 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881238 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.887141 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881241 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881243 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881246 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881249 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881251 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881254 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881257 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881259 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881262 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881264 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881267 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881270 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881273 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881277 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881281 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881285 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881288 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881291 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881300 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.887632 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881303 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881305 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881308 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881311 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881313 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881316 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881318 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881320 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881323 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881325 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881328 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881330 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881333 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881335 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881338 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881340 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881343 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881345 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881348 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.888163 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881350 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881353 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881355 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881358 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881360 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881363 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881365 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881367 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881370 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881374 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881379 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881383 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881385 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881394 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881397 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881399 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881401 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881404 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881406 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.888641 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881409 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881412 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881415 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881417 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881420 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881423 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881426 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881429 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881431 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881434 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881436 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881439 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881441 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881444 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881446 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881449 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881451 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881454 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881456 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881459 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.889368 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881461 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881464 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881467 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881469 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881472 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881475 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881477 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881485 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881488 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881491 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881493 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881496 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881498 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881501 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881503 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881506 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881508 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.890126 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881511 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.881541 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881732 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881737 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881754 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881758 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881761 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881764 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881769 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881772 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881774 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881777 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881780 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881784 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881787 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.890594 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881790 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881793 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881796 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881799 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881801 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881804 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881806 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881816 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881819 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881822 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881824 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881827 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881829 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881832 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881834 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881837 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881839 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881842 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881845 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.891032 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881847 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881850 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881853 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881855 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881858 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881860 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881863 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881865 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881867 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881870 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881872 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881875 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881877 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881881 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881885 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881888 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881892 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881895 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881897 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.891524 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881900 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881903 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881912 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881915 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881917 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881920 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881922 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881925 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881927 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881930 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881932 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881935 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881938 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881940 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881943 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881945 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881947 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881950 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881953 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881955 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.892183 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881957 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881960 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881962 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881971 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881973 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881976 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881978 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881981 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881986 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881989 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881991 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881993 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881996 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.881998 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882001 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882009 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882012 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882015 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882018 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.892692 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882021 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882024 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882026 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882029 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882031 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882034 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882037 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882039 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882042 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882044 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882047 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882049 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882052 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882055 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882058 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.893315 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.882082 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882120 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882125 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882128 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882131 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882133 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882136 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882140 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882143 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882146 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882148 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882151 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882154 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882156 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882165 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882169 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882173 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882176 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882179 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:33:39.893830 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882182 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882185 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882187 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882190 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882192 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882195 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882198 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882200 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882203 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882206 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882208 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882211 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882213 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882216 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882218 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882221 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882224 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882227 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882229 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:33:39.894269 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882232 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882237 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882239 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882242 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882244 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882247 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882249 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882252 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882254 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882263 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882265 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882269 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882273 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882277 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882281 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882283 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882286 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882288 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882291 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:33:39.894715 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882294 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882296 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882299 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882301 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882304 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882306 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882309 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882311 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882314 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882317 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882320 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882322 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882325 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882327 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882331 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882334 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882337 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882339 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882342 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882344 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:33:39.895316 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882347 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882349 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882352 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882361 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882365 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882368 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882370 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882373 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882376 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882378 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882381 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882383 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882386 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882389 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882391 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882394 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882396 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882399 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882401 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882404 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:33:39.895847 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882406 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882409 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882411 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882414 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882416 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882419 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882423 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882425 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882428 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:39.882431 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.882489 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.885114 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.886030 2580 server.go:1014] "Starting client certificate rotation" Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.886139 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.886185 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.888009 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:33:39.896319 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.894782 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:33:39.904810 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.904765 2580 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:33:39.907844 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.907829 2580 log.go:25] "Validated CRI v1 image API" Nov 02 21:33:39.908665 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.908637 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:33:39.912496 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.912474 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a7ab3d2a-a85d-47cf-b1d9-e7c412a8210e:/dev/nvme0n1p3 d088ad82-2b06-4530-becd-c3a69f58ebf6:/dev/nvme0n1p4] Nov 02 21:33:39.912586 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.912495 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:33:39.917976 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.917856 2580 manager.go:217] Machine: {Timestamp:2026-11-02 21:33:39.916453178 +0000 UTC m=+0.313094057 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499998 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229189611e0c7efb84ea60d137be64 SystemUUID:ec229189-611e-0c7e-fb84-ea60d137be64 BootID:34d7e761-90cb-433c-9d1e-6431e3cb858c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:69:af:82:d2:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:69:af:82:d2:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:67:91:54:49:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:33:39.917976 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.917969 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:33:39.918161 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.918080 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:33:39.919276 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.919249 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:33:39.919403 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.919277 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-47.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:33:39.919444 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.919406 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:33:39.919444 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.919415 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:33:39.919508 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.919446 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:33:39.919508 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.919464 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:33:39.920215 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.920204 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:33:39.923752 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.923731 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:33:39.923873 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.923864 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:33:39.924580 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.924571 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:33:39.924607 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.924584 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:33:39.924607 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.924595 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:33:39.924607 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.924604 2580 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:33:39.924717 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.924613 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:33:39.925016 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.925005 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:33:39.927223 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.927198 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:33:39.931728 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.931716 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933366 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933427 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933651 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933668 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933679 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933690 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933708 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:33:39.933757 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.933718 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:33:39.934273 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.934251 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:33:39.934336 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.934277 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:33:39.934336 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.934294 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:33:39.934336 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.934304 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:33:39.934336 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.934322 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:33:39.934336 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.934335 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:33:39.935207 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.935188 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:33:39.935257 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.935212 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:33:39.939823 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.939807 2580 server.go:1263] "Started kubelet" Nov 02 21:33:39.939911 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.939883 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:33:39.939957 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.939896 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:33:39.939994 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.939980 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:33:39.940556 ip-10-0-131-47 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:33:39.941736 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.940943 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:33:39.941904 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.941885 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:33:39.945328 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.945303 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:33:39.945328 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.945320 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:33:39.945470 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:39.945393 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:33:39.945470 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:39.945430 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:39.945863 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.945848 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:33:39.946105 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946087 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:33:39.946172 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946108 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:33:39.946257 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946243 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:33:39.946306 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946258 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:33:39.946590 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946570 2580 factory.go:55] Registering systemd factory Nov 02 21:33:39.946590 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946589 2580 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:33:39.946847 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946830 2580 factory.go:153] Registering CRI-O factory Nov 02 21:33:39.946953 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946851 2580 factory.go:223] Registration of the crio container factory successfully Nov 02 21:33:39.946953 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946925 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:33:39.946953 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946944 2580 factory.go:103] Registering Raw factory Nov 02 21:33:39.947091 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.946959 2580 manager.go:1199] Started watching for new ooms in manager Nov 02 21:33:39.947846 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.947829 2580 manager.go:319] Starting recovery of all containers Nov 02 21:33:39.958420 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.958405 2580 manager.go:324] Recovery completed Nov 02 21:33:39.963394 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.963378 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:39.965053 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965007 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:39.965053 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965038 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:39.965053 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965051 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:39.965615 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965599 2580 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:33:39.965615 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965613 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:33:39.966293 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965634 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:33:39.966293 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.965642 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:39.967399 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.967382 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:39.967475 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.967415 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:39.967475 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.967429 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:39.968034 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.968019 2580 policy_none.go:50] "Start" Nov 02 21:33:39.968080 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.968041 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:33:39.968080 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.968058 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:33:39.970286 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.970269 2580 policy_none.go:44] "Start" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.995234 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.998483 2580 manager.go:341] "Starting Device Plugin manager" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:39.998520 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.998533 2580 server.go:87] "Starting device plugin registration server" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.998732 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.998756 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.999152 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.999218 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:39.999223 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:39.999735 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:39.999810 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.009500 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.010573 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.010914 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.010941 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:33:40.017251 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.010978 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:33:40.083285 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.083235 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-47.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:33:40.083285 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.083231 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-47.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:33:40.084300 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.083279 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-47.us-west-1.compute.internal.18e457eb5a091824 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-47.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-47.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 21:33:39.939776548 +0000 UTC m=+0.336417426,LastTimestamp:2026-11-02 21:33:39.939776548 +0000 UTC m=+0.336417426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-47.us-west-1.compute.internal,}" Nov 02 21:33:40.099911 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.099889 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:40.100873 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.100857 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:40.100954 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.100886 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:40.100954 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.100898 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:40.100954 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.100924 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.111832 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.111810 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal"] Nov 02 21:33:40.111874 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.111865 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:40.112677 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.112661 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:40.112809 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.112692 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:40.112809 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.112703 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:40.115068 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.114965 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:40.115157 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.115157 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115150 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:40.115756 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115720 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:40.115839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115769 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:40.115839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115781 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:40.115839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115805 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:40.115839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115818 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:40.115839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.115784 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:40.118029 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.118014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.118093 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.118046 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:40.118731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.118715 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:40.118839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.118755 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:40.118839 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.118773 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:40.147355 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.147181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef8aa625af233e1fa89cb16cb4aca78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"ef8aa625af233e1fa89cb16cb4aca78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.147437 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.147369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.147437 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.147389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.154397 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.154378 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.159828 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.159809 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.165443 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.165418 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 21:33:40.165533 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.165514 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 21:33:40.210110 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.210085 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:33:40.210211 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.210147 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7mltn" Nov 02 21:33:40.214430 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.214410 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 21:33:40.214430 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.214365 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-47.us-west-1.compute.internal.18e457eb5b8a514d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-47.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-47.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-47.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 21:33:39.965022541 +0000 UTC m=+0.361663419,LastTimestamp:2026-11-02 21:33:39.965022541 +0000 UTC m=+0.361663419,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-47.us-west-1.compute.internal,}" Nov 02 21:33:40.214529 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.214435 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-47.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 21:33:40.237241 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.237217 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-47.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.247682 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.247639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.247761 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.247688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.247761 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.247706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef8aa625af233e1fa89cb16cb4aca78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"ef8aa625af233e1fa89cb16cb4aca78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.247835 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.247770 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef8aa625af233e1fa89cb16cb4aca78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"ef8aa625af233e1fa89cb16cb4aca78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.247835 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.247775 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.247835 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.247775 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.386049 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.385965 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-47.us-west-1.compute.internal\" not found" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.438088 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.438062 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:40.439089 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.439076 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:40.439154 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.439106 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:40.439154 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.439120 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:40.439263 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.439156 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.456519 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.456500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.463995 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.463982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.494998 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:40.494979 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:40.495085 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.495001 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-47.us-west-1.compute.internal\": node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:40.612062 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.612027 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:40.712195 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.712103 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:40.812365 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:40.812335 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:41.968787 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:41.968760 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:33:41.969254 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:41.968877 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a844a01bd9c1a45928f36aeb9963b228-b8a9b0af8073bf35.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-47.us-west-1.compute.internal.18e457eb5b8a514d\": read tcp 10.0.131.47:49248->3.130.5.189:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-47.us-west-1.compute.internal.18e457eb5b8a514d default 5075 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-47.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-47.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-47.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 21:33:39 +0000 UTC,LastTimestamp:2026-11-02 21:33:40.112677081 +0000 UTC m=+0.509317959,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-47.us-west-1.compute.internal,}" Nov 02 21:33:41.995070 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:41.995045 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.095247 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:42.095184 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.195759 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:42.195731 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.239586 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.239560 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:33:42.240631 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.240611 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:33:42.240713 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.240645 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:33:42.240713 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.240656 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:33:42.296234 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:42.296202 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.297290 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.297270 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:28:41 +0000 UTC" deadline="2028-07-02 07:57:07.393187726 +0000 UTC" Nov 02 21:33:42.297338 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.297291 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14578h23m25.095899369s" Nov 02 21:33:42.396729 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:42.396655 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.423093 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.423067 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:42.423232 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.423109 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:42.497240 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:42.497193 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.598312 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:42.598260 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:33:42.626965 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:42.626934 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8aa625af233e1fa89cb16cb4aca78e.slice/crio-64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935 WatchSource:0}: Error finding container 64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935: Status 404 returned error can't find the container with id 64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935 Nov 02 21:33:42.627162 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:42.627139 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d7a9d6fa496da9c68538ee1b34b8e7.slice/crio-7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df WatchSource:0}: Error finding container 7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df: Status 404 returned error can't find the container with id 7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df Nov 02 21:33:42.633029 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.633012 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:33:42.633107 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.633075 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:42.728549 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.728481 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:42.787470 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.787446 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:33:42.788472 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.788459 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:33:42.845415 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.845387 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:33:42.924378 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:42.924351 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:33:43.008092 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.008020 2580 apiserver.go:52] "Watching apiserver" Nov 02 21:33:43.029251 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.029222 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:33:43.098788 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.098741 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"c9d7a9d6fa496da9c68538ee1b34b8e7","Type":"ContainerStarted","Data":"7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df"} Nov 02 21:33:43.099714 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.099691 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"ef8aa625af233e1fa89cb16cb4aca78e","Type":"ContainerStarted","Data":"64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935"} Nov 02 21:33:43.109538 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.109522 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:33:43.109826 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.109811 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal","openshift-dns/node-resolver-gkphg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-k75zj","openshift-image-registry/node-ca-9wp2q","openshift-ovn-kubernetes/ovnkube-node-8wg6d","kube-system/global-pull-secret-syncer-lmrxw","openshift-cluster-node-tuning-operator/tuned-srw9t","openshift-multus/multus-additional-cni-plugins-mv42m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf","openshift-multus/multus-8zxht","openshift-network-diagnostics/network-check-target-gbwsv","openshift-network-operator/iptables-alerter-btpw2","kube-system/konnectivity-agent-rtnjt"] Nov 02 21:33:43.113685 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.113667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.113771 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.113751 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:43.115714 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.115701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.117729 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.117712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.119796 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.119784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.119850 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.119833 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:43.121879 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.121857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.124149 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.124121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.126475 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.126455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.128747 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.128730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.130941 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.130925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.133379 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.133362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.137233 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.137200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.138856 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2615d298-b340-465b-b9b0-f66aa0f59628-agent-certs\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.138946 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-etc-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.138946 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-log-socket\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.138946 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.138946 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139108 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddrs\" (UniqueName: \"kubernetes.io/projected/295c327d-c476-4f98-84a1-a39dda118c73-kube-api-access-fddrs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.139108 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.138994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-systemd-units\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139108 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-var-lib-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139108 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysconfig\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139108 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-var-lib-kubelet\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-kubelet\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-bin\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovn-node-metrics-cert\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-system-cni-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e761f1b3-b683-463e-9c05-6432569d4cd0-host-slash\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-dbus\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.139366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-kubelet-config\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-modprobe-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-run\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.139417 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-conf\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-tmp\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e761f1b3-b683-463e-9c05-6432569d4cd0-iptables-alerter-script\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46x6\" (UniqueName: \"kubernetes.io/projected/e761f1b3-b683-463e-9c05-6432569d4cd0-kube-api-access-n46x6\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2615d298-b340-465b-b9b0-f66aa0f59628-konnectivity-ca\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf29\" (UniqueName: \"kubernetes.io/projected/4321e24b-79fa-4817-a0eb-66f3d9c875be-kube-api-access-rzf29\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-sys\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-etc-tuned\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.139731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-cnibin\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-systemd\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-script-lib\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da3003ef-ed3e-40ea-8dec-781b93a27cc2-serviceca\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-host\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-netns\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-env-overrides\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-kubernetes\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8d2\" (UniqueName: \"kubernetes.io/projected/32d0f122-5c7a-45b3-868e-03157b31421a-kube-api-access-5t8d2\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-slash\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-node-log\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-config\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8sjh\" (UniqueName: \"kubernetes.io/projected/da3003ef-ed3e-40ea-8dec-781b93a27cc2-kube-api-access-b8sjh\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-lib-modules\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.139995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-systemd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.140016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.140540 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.140030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da3003ef-ed3e-40ea-8dec-781b93a27cc2-host\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.141380 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.140056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-ovn\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.141380 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.140074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv79b\" (UniqueName: \"kubernetes.io/projected/3a1e59cc-2f43-4079-9713-b19528741a1c-kube-api-access-hv79b\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.141380 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.140091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-netd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.141380 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.140107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.167424 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.167400 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.167519 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.167450 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nr5z4\"" Nov 02 21:33:43.167519 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.167484 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:33:43.167616 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.167546 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.168772 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.168756 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:33:43.168957 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.168861 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b8lsz\"" Nov 02 21:33:43.168957 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.168928 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:33:43.173214 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.173196 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.173359 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.173344 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.173401 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.173369 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:33:43.173401 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.173385 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sb4td\"" Nov 02 21:33:43.175301 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.175288 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsxz6\"" Nov 02 21:33:43.175490 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.175478 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:33:43.175529 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.175504 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:33:43.175748 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.175735 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.175865 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.175852 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:33:43.175865 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.175861 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.176177 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.176159 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:33:43.177547 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.177531 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:33:43.177599 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.177552 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.177599 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.177566 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.177599 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.177587 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qbt7b\"" Nov 02 21:33:43.177725 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.177604 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:33:43.177725 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.177635 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:33:43.179691 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.179677 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.179845 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.179833 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.179884 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.179844 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95jdp\"" Nov 02 21:33:43.210615 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210591 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:33:43.210728 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210645 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.210728 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210651 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.210728 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210657 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2mch2\"" Nov 02 21:33:43.210728 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210719 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:33:43.210901 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210736 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:33:43.210901 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210782 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7clxp\"" Nov 02 21:33:43.210901 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210781 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f7c29\"" Nov 02 21:33:43.210901 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.210808 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:33:43.212063 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.212050 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:33:43.212104 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.212080 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sh5vq" Nov 02 21:33:43.229058 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.229041 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:33:43.240705 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240685 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-systemd-units\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.240773 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-systemd-units\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.240773 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-var-lib-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.240773 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-var-lib-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.240854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysconfig\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.240854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-var-lib-kubelet\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.240854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-kubelet\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.240854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-bin\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovn-node-metrics-cert\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-system-cni-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-var-lib-kubelet\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240920 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-kubelet\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240930 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysconfig\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.241027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.240947 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-bin\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241049 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-system-cni-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e761f1b3-b683-463e-9c05-6432569d4cd0-host-slash\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-dbus\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241168 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e761f1b3-b683-463e-9c05-6432569d4cd0-host-slash\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-system-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-netns\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.241352 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241272 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:33:43.241625 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241276 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-dbus\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.241694 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.241679 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:43.241694 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-daemon-config\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.241795 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.241779 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret podName:befb271e-da36-470a-ae6e-b7ce5dabf394 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:43.741751368 +0000 UTC m=+3.056010867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret") pod "global-pull-secret-syncer-lmrxw" (UID: "befb271e-da36-470a-ae6e-b7ce5dabf394") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:43.241862 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-kubelet-config\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.241862 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241821 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-kubelet-config\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.241862 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-modprobe-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.241862 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-run\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-tmp-dir\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-device-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241891 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-modprobe-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-cni-binary-copy\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-kubelet\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241927 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-conf\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-tmp\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.241915 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-run\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.242024 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e761f1b3-b683-463e-9c05-6432569d4cd0-iptables-alerter-script\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242086 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-conf\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46x6\" (UniqueName: \"kubernetes.io/projected/e761f1b3-b683-463e-9c05-6432569d4cd0-kube-api-access-n46x6\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2615d298-b340-465b-b9b0-f66aa0f59628-konnectivity-ca\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf29\" (UniqueName: \"kubernetes.io/projected/4321e24b-79fa-4817-a0eb-66f3d9c875be-kube-api-access-rzf29\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.242259 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-sys\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.242307 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:43.742289421 +0000 UTC m=+3.056548906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242312 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-sys\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-etc-tuned\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.242535 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-multus\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-conf-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-etc-kubernetes\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-cnibin\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242583 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e761f1b3-b683-463e-9c05-6432569d4cd0-iptables-alerter-script\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-systemd\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242667 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-cnibin\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242738 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-systemd\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242743 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-script-lib\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242743 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2615d298-b340-465b-b9b0-f66aa0f59628-konnectivity-ca\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da3003ef-ed3e-40ea-8dec-781b93a27cc2-serviceca\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-hosts-file\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-socket-dir-parent\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.243240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242948 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.242955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-host\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-netns\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243029 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-host\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243037 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-netns\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-env-overrides\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-kubernetes\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8d2\" (UniqueName: \"kubernetes.io/projected/32d0f122-5c7a-45b3-868e-03157b31421a-kube-api-access-5t8d2\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-slash\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-node-log\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243239 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da3003ef-ed3e-40ea-8dec-781b93a27cc2-serviceca\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-config\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243312 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-script-lib\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243360 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-kubernetes\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243389 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-slash\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sjh\" (UniqueName: \"kubernetes.io/projected/da3003ef-ed3e-40ea-8dec-781b93a27cc2-kube-api-access-b8sjh\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243437 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-node-log\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-lib-modules\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244001 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-k8s-cni-cncf-io\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4tv\" (UniqueName: \"kubernetes.io/projected/cfd48ab2-58a6-42c2-8077-3b9af1669069-kube-api-access-fq4tv\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-lib-modules\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243512 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-env-overrides\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-systemd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243566 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-systemd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvqd\" (UniqueName: \"kubernetes.io/projected/cd749222-a1c6-403b-bcfe-4897707b96cc-kube-api-access-zlvqd\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-bin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-hostroot\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da3003ef-ed3e-40ea-8dec-781b93a27cc2-host\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-ovn\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243701 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv79b\" (UniqueName: \"kubernetes.io/projected/3a1e59cc-2f43-4079-9713-b19528741a1c-kube-api-access-hv79b\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243753 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-ovn\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-cnibin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-multus-certs\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.244828 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da3003ef-ed3e-40ea-8dec-781b93a27cc2-host\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-netd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-netd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgf2\" (UniqueName: \"kubernetes.io/projected/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-kube-api-access-mlgf2\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.243974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2615d298-b340-465b-b9b0-f66aa0f59628-agent-certs\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-etc-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244077 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-etc-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-log-socket\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244158 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-log-socket\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddrs\" (UniqueName: \"kubernetes.io/projected/295c327d-c476-4f98-84a1-a39dda118c73-kube-api-access-fddrs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-config\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244212 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245350 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.244306 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.245854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.245086 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-tmp\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.245854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.245101 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-etc-tuned\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.245854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.245159 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovn-node-metrics-cert\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.245854 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.245249 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.246203 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.246188 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2615d298-b340-465b-b9b0-f66aa0f59628-agent-certs\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.298777 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.298700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46x6\" (UniqueName: \"kubernetes.io/projected/e761f1b3-b683-463e-9c05-6432569d4cd0-kube-api-access-n46x6\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.299140 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.299091 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8d2\" (UniqueName: \"kubernetes.io/projected/32d0f122-5c7a-45b3-868e-03157b31421a-kube-api-access-5t8d2\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.299140 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.299117 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf29\" (UniqueName: \"kubernetes.io/projected/4321e24b-79fa-4817-a0eb-66f3d9c875be-kube-api-access-rzf29\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.299383 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.299365 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sjh\" (UniqueName: \"kubernetes.io/projected/da3003ef-ed3e-40ea-8dec-781b93a27cc2-kube-api-access-b8sjh\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.300033 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.300011 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv79b\" (UniqueName: \"kubernetes.io/projected/3a1e59cc-2f43-4079-9713-b19528741a1c-kube-api-access-hv79b\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.300368 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.300352 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddrs\" (UniqueName: \"kubernetes.io/projected/295c327d-c476-4f98-84a1-a39dda118c73-kube-api-access-fddrs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.344836 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgf2\" (UniqueName: \"kubernetes.io/projected/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-kube-api-access-mlgf2\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.344836 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345031 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345031 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345031 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345031 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-system-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345234 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345057 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-netns\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345348 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345326 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345416 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345191 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345416 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-netns\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345416 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345400 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345557 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-daemon-config\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345557 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345424 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-system-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345557 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-tmp-dir\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.345557 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.344950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345557 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-device-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345783 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345578 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-device-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.345783 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-cni-binary-copy\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345783 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-kubelet\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345783 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345753 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345785 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-tmp-dir\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345792 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-kubelet\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-multus\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-conf-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345970 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-daemon-config\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.345978 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.345978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-etc-kubernetes\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346040 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-conf-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346080 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-multus\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-hosts-file\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-socket-dir-parent\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-k8s-cni-cncf-io\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346185 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-cni-binary-copy\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4tv\" (UniqueName: \"kubernetes.io/projected/cfd48ab2-58a6-42c2-8077-3b9af1669069-kube-api-access-fq4tv\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvqd\" (UniqueName: \"kubernetes.io/projected/cd749222-a1c6-403b-bcfe-4897707b96cc-kube-api-access-zlvqd\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346280 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-socket-dir-parent\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-bin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346326 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-hostroot\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-cnibin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-k8s-cni-cncf-io\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346393 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-hostroot\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346404 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-cnibin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346326 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346657 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-hosts-file\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-multus-certs\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346715 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-etc-kubernetes\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346771 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-multus-certs\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.346905 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.346807 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-bin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.399295 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.399270 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:43.399295 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.399294 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:43.399295 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.399305 2580 projected.go:196] Error preparing data for projected volume kube-api-access-j626w for pod openshift-network-diagnostics/network-check-target-gbwsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:43.399544 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.399372 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w podName:92bc027f-95b3-4c18-96b0-9d7bd6ab4413 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:43.89935822 +0000 UTC m=+3.213617702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j626w" (UniqueName: "kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w") pod "network-check-target-gbwsv" (UID: "92bc027f-95b3-4c18-96b0-9d7bd6ab4413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:43.401356 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.401328 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgf2\" (UniqueName: \"kubernetes.io/projected/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-kube-api-access-mlgf2\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.401471 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.401454 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4tv\" (UniqueName: \"kubernetes.io/projected/cfd48ab2-58a6-42c2-8077-3b9af1669069-kube-api-access-fq4tv\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.401597 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.401579 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvqd\" (UniqueName: \"kubernetes.io/projected/cd749222-a1c6-403b-bcfe-4897707b96cc-kube-api-access-zlvqd\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.427428 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.427405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:33:43.432695 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.432674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:43.439812 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.439791 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode761f1b3_b683_463e_9c05_6432569d4cd0.slice/crio-f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422 WatchSource:0}: Error finding container f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422: Status 404 returned error can't find the container with id f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422 Nov 02 21:33:43.441362 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.441283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:33:43.442666 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.442648 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2615d298_b340_465b_b9b0_f66aa0f59628.slice/crio-5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d WatchSource:0}: Error finding container 5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d: Status 404 returned error can't find the container with id 5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d Nov 02 21:33:43.450092 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.450071 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3003ef_ed3e_40ea_8dec_781b93a27cc2.slice/crio-c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a WatchSource:0}: Error finding container c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a: Status 404 returned error can't find the container with id c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a Nov 02 21:33:43.463058 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.463040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:33:43.466529 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.466514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkphg" Nov 02 21:33:43.470862 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.470842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:33:43.474366 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.474345 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d0f122_5c7a_45b3_868e_03157b31421a.slice/crio-e72edc165414d3025bc50e9d506100a2afb183e985b7ca2bc5b3942c5549d7b4 WatchSource:0}: Error finding container e72edc165414d3025bc50e9d506100a2afb183e985b7ca2bc5b3942c5549d7b4: Status 404 returned error can't find the container with id e72edc165414d3025bc50e9d506100a2afb183e985b7ca2bc5b3942c5549d7b4 Nov 02 21:33:43.479004 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.478983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8zxht" Nov 02 21:33:43.480585 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.480566 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58637e1f_1378_4b0a_9c11_03f3fe45d6ae.slice/crio-c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c WatchSource:0}: Error finding container c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c: Status 404 returned error can't find the container with id c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c Nov 02 21:33:43.483295 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.483272 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4321e24b_79fa_4817_a0eb_66f3d9c875be.slice/crio-db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0 WatchSource:0}: Error finding container db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0: Status 404 returned error can't find the container with id db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0 Nov 02 21:33:43.484328 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.484311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:33:43.486864 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.486839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:33:43.488860 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.488836 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd48ab2_58a6_42c2_8077_3b9af1669069.slice/crio-2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984 WatchSource:0}: Error finding container 2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984: Status 404 returned error can't find the container with id 2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984 Nov 02 21:33:43.495460 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.495430 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1e59cc_2f43_4079_9713_b19528741a1c.slice/crio-24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b WatchSource:0}: Error finding container 24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b: Status 404 returned error can't find the container with id 24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b Nov 02 21:33:43.497360 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:33:43.497326 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd749222_a1c6_403b_bcfe_4897707b96cc.slice/crio-843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4 WatchSource:0}: Error finding container 843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4: Status 404 returned error can't find the container with id 843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4 Nov 02 21:33:43.749775 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.749693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:43.749775 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.749747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:43.749999 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.749854 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:43.749999 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.749925 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret podName:befb271e-da36-470a-ae6e-b7ce5dabf394 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:44.749908297 +0000 UTC m=+4.064167793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret") pod "global-pull-secret-syncer-lmrxw" (UID: "befb271e-da36-470a-ae6e-b7ce5dabf394") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:43.749999 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.749859 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:43.749999 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.749984 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:44.749971968 +0000 UTC m=+4.064231452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:43.951546 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:43.951500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:43.951799 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.951637 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:43.951799 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.951655 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:43.951799 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.951669 2580 projected.go:196] Error preparing data for projected volume kube-api-access-j626w for pod openshift-network-diagnostics/network-check-target-gbwsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:43.951799 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:43.951725 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w podName:92bc027f-95b3-4c18-96b0-9d7bd6ab4413 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:44.95170883 +0000 UTC m=+4.265968329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j626w" (UniqueName: "kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w") pod "network-check-target-gbwsv" (UID: "92bc027f-95b3-4c18-96b0-9d7bd6ab4413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:44.103027 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.102939 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-srw9t" event={"ID":"32d0f122-5c7a-45b3-868e-03157b31421a","Type":"ContainerStarted","Data":"e72edc165414d3025bc50e9d506100a2afb183e985b7ca2bc5b3942c5549d7b4"} Nov 02 21:33:44.104232 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.104198 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-btpw2" event={"ID":"e761f1b3-b683-463e-9c05-6432569d4cd0","Type":"ContainerStarted","Data":"f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422"} Nov 02 21:33:44.105591 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.105562 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" event={"ID":"cd749222-a1c6-403b-bcfe-4897707b96cc","Type":"ContainerStarted","Data":"843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4"} Nov 02 21:33:44.106857 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.106819 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerStarted","Data":"24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b"} Nov 02 21:33:44.108077 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.108046 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0"} Nov 02 21:33:44.109518 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.109497 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wp2q" event={"ID":"da3003ef-ed3e-40ea-8dec-781b93a27cc2","Type":"ContainerStarted","Data":"c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a"} Nov 02 21:33:44.110766 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.110745 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rtnjt" event={"ID":"2615d298-b340-465b-b9b0-f66aa0f59628","Type":"ContainerStarted","Data":"5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d"} Nov 02 21:33:44.111968 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.111944 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8zxht" event={"ID":"cfd48ab2-58a6-42c2-8077-3b9af1669069","Type":"ContainerStarted","Data":"2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984"} Nov 02 21:33:44.113216 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.113197 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkphg" event={"ID":"58637e1f-1378-4b0a-9c11-03f3fe45d6ae","Type":"ContainerStarted","Data":"c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c"} Nov 02 21:33:44.215589 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.215536 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:28:43 +0000 UTC" deadline="2028-05-04 15:22:59.728456731 +0000 UTC" Nov 02 21:33:44.215589 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.215583 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13169h49m15.512878834s" Nov 02 21:33:44.759521 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.759441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:44.759674 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.759563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:44.759789 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.759771 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:44.759854 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.759838 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:46.759820586 +0000 UTC m=+6.074080071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:44.759939 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.759926 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:44.759990 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.759966 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret podName:befb271e-da36-470a-ae6e-b7ce5dabf394 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:46.75995442 +0000 UTC m=+6.074213908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret") pod "global-pull-secret-syncer-lmrxw" (UID: "befb271e-da36-470a-ae6e-b7ce5dabf394") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:44.963797 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:44.963743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:44.963985 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.963972 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:44.964045 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.963996 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:44.964045 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.964009 2580 projected.go:196] Error preparing data for projected volume kube-api-access-j626w for pod openshift-network-diagnostics/network-check-target-gbwsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:44.964152 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:44.964068 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w podName:92bc027f-95b3-4c18-96b0-9d7bd6ab4413 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:46.96404944 +0000 UTC m=+6.278308924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j626w" (UniqueName: "kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w") pod "network-check-target-gbwsv" (UID: "92bc027f-95b3-4c18-96b0-9d7bd6ab4413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:45.097276 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:45.097210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:45.097422 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:45.097341 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:45.097486 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:45.097429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:45.097534 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:45.097510 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:45.097583 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:45.097567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:45.097658 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:45.097640 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:45.216104 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:45.216039 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:28:43 +0000 UTC" deadline="2028-07-24 05:22:46.425190632 +0000 UTC" Nov 02 21:33:45.216104 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:45.216102 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15103h49m1.209093144s" Nov 02 21:33:46.128877 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:46.128020 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9d7a9d6fa496da9c68538ee1b34b8e7" containerID="253d60cdea4799fe449cf3f3231013836cc6d0da0150054c9ff960b3aa52211d" exitCode=0 Nov 02 21:33:46.128877 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:46.128201 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"c9d7a9d6fa496da9c68538ee1b34b8e7","Type":"ContainerDied","Data":"253d60cdea4799fe449cf3f3231013836cc6d0da0150054c9ff960b3aa52211d"} Nov 02 21:33:46.138479 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:46.137998 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"ef8aa625af233e1fa89cb16cb4aca78e","Type":"ContainerStarted","Data":"d6efb358e248d309b0195e01ff9e88e85613f8456828a39bb1c0a3404934f107"} Nov 02 21:33:46.781044 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:46.780998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:46.781528 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:46.781061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:46.781528 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.781227 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:46.781528 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.781292 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:50.781272988 +0000 UTC m=+10.095532476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:46.781528 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.781364 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:46.781528 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.781401 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret podName:befb271e-da36-470a-ae6e-b7ce5dabf394 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:50.781388409 +0000 UTC m=+10.095647896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret") pod "global-pull-secret-syncer-lmrxw" (UID: "befb271e-da36-470a-ae6e-b7ce5dabf394") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:46.982692 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:46.982609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:46.982839 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.982798 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:46.982839 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.982821 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:46.982839 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.982834 2580 projected.go:196] Error preparing data for projected volume kube-api-access-j626w for pod openshift-network-diagnostics/network-check-target-gbwsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:46.982996 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:46.982896 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w podName:92bc027f-95b3-4c18-96b0-9d7bd6ab4413 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:50.982878273 +0000 UTC m=+10.297137774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j626w" (UniqueName: "kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w") pod "network-check-target-gbwsv" (UID: "92bc027f-95b3-4c18-96b0-9d7bd6ab4413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:47.096356 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.096138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:47.096356 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.096302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:47.096356 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:47.096291 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:47.096678 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:47.096404 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:47.096678 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.096446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:47.096678 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:47.096572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:47.147145 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.145296 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"c9d7a9d6fa496da9c68538ee1b34b8e7","Type":"ContainerStarted","Data":"b713314f30527f151e8bb45d8bc2caf1c6c511aaaee524522e28b8f1ece6b211"} Nov 02 21:33:47.148495 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.148463 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="89e060b6c17124ef2e02d179f3baf035d789dfdd08dc3f55416141c55354d52f" exitCode=0 Nov 02 21:33:47.149466 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.149442 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerDied","Data":"89e060b6c17124ef2e02d179f3baf035d789dfdd08dc3f55416141c55354d52f"} Nov 02 21:33:47.258575 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.257600 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" podStartSLOduration=5.257581615 podStartE2EDuration="5.257581615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:46.351506174 +0000 UTC m=+5.665765679" watchObservedRunningTime="2026-11-02 21:33:47.257581615 +0000 UTC m=+6.571841121" Nov 02 21:33:47.374558 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:47.373488 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" podStartSLOduration=5.373468809 podStartE2EDuration="5.373468809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:47.258072358 +0000 UTC m=+6.572331857" watchObservedRunningTime="2026-11-02 21:33:47.373468809 +0000 UTC m=+6.687728317" Nov 02 21:33:48.156805 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:48.156727 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rtnjt" event={"ID":"2615d298-b340-465b-b9b0-f66aa0f59628","Type":"ContainerStarted","Data":"3c191b4bf44159e3cd9402d130cd4b97f1585395b1642f7d4b60746eb7248b49"} Nov 02 21:33:48.267675 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:48.267613 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rtnjt" podStartSLOduration=2.915072958 podStartE2EDuration="7.267590971s" totalImagesPullingTime="4.352518013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:48.266667323 +0000 UTC m=+7.580926827" watchObservedRunningTime="2026-11-02 21:33:48.267590971 +0000 UTC m=+7.581850480" Nov 02 21:33:49.095468 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:49.095003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:49.095468 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:49.095149 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:49.095468 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:49.095193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:49.095468 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:49.095277 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:49.095468 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:49.095334 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:49.095468 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:49.095403 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:50.166997 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.166907 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" event={"ID":"cd749222-a1c6-403b-bcfe-4897707b96cc","Type":"ContainerStarted","Data":"5906a1b4186661c073744113e6f3539e2f4185cdea366966212918eb5d2fdc98"} Nov 02 21:33:50.169643 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.169609 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wp2q" event={"ID":"da3003ef-ed3e-40ea-8dec-781b93a27cc2","Type":"ContainerStarted","Data":"56efa06f9029ed627554605954af89787008d9770bc566c6794b3a3ce9c8ee65"} Nov 02 21:33:50.756399 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.755678 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:50.757032 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.756814 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:50.814370 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.814337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:50.814548 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.814413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:50.814876 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:50.814854 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:50.814936 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:50.814920 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret podName:befb271e-da36-470a-ae6e-b7ce5dabf394 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:58.814902292 +0000 UTC m=+18.129161788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret") pod "global-pull-secret-syncer-lmrxw" (UID: "befb271e-da36-470a-ae6e-b7ce5dabf394") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:50.815307 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:50.815278 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:50.815362 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:50.815349 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:58.81533328 +0000 UTC m=+18.129592769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:50.867969 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:50.867899 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wp2q" podStartSLOduration=4.322682995 podStartE2EDuration="9.867878105s" totalImagesPullingTime="5.54519511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:50.281302572 +0000 UTC m=+9.595562077" watchObservedRunningTime="2026-11-02 21:33:50.867878105 +0000 UTC m=+10.182137610" Nov 02 21:33:51.018736 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:51.018657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:51.018903 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.018851 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:51.018903 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.018870 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:51.018903 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.018882 2580 projected.go:196] Error preparing data for projected volume kube-api-access-j626w for pod openshift-network-diagnostics/network-check-target-gbwsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:51.019079 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.018938 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w podName:92bc027f-95b3-4c18-96b0-9d7bd6ab4413 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:59.018919447 +0000 UTC m=+18.333178944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j626w" (UniqueName: "kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w") pod "network-check-target-gbwsv" (UID: "92bc027f-95b3-4c18-96b0-9d7bd6ab4413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:51.097016 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:51.096986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:51.097409 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:51.097349 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:51.097512 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.097457 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:51.097577 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:51.097526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:51.097625 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.097605 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:51.097678 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:51.097106 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:51.531299 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:51.531266 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:33:52.087972 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.087866 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:33:51.53128221Z","UUID":"a4fc24f6-1f30-48d8-968d-b85c575b7497","Handler":null,"Name":"","Endpoint":""} Nov 02 21:33:52.089549 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.089527 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:33:52.089678 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.089564 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:33:52.176262 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.176227 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" event={"ID":"cd749222-a1c6-403b-bcfe-4897707b96cc","Type":"ContainerStarted","Data":"6fe1b9ed0ebe06c42f863364ce29ef57ea62bc7519e34dde8a7227c1d47d36f7"} Nov 02 21:33:52.179056 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.179025 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerStarted","Data":"52515717d0160ba3c28f8085fd2946802ac42a5a017e904e38d3aede15b20815"} Nov 02 21:33:52.181107 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.181085 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8zxht" event={"ID":"cfd48ab2-58a6-42c2-8077-3b9af1669069","Type":"ContainerStarted","Data":"3ea28fab3b88fb60de272d8c2881e615971b6e4f3cd66c7b0179a47853149d98"} Nov 02 21:33:52.182744 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.182721 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkphg" event={"ID":"58637e1f-1378-4b0a-9c11-03f3fe45d6ae","Type":"ContainerStarted","Data":"593a8e933e19a68eba1bc0ffb4b0ef2f74ee67e8863ca3edcd7f3e23e98faeb1"} Nov 02 21:33:52.510052 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.509975 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:52.510213 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.510184 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:33:52.510761 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.510697 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:33:52.531670 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.531593 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8zxht" podStartSLOduration=3.439067473 podStartE2EDuration="11.531573585s" totalImagesPullingTime="8.092506112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:52.420930188 +0000 UTC m=+11.735189693" watchObservedRunningTime="2026-11-02 21:33:52.531573585 +0000 UTC m=+11.845833090" Nov 02 21:33:52.532152 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:52.531936 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gkphg" podStartSLOduration=3.162455228 podStartE2EDuration="11.531923806s" totalImagesPullingTime="8.369468578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:52.531110765 +0000 UTC m=+11.845370270" watchObservedRunningTime="2026-11-02 21:33:52.531923806 +0000 UTC m=+11.846183315" Nov 02 21:33:53.097366 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.097081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:53.097546 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:53.097509 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:53.097607 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.097589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:53.097679 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:53.097661 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:53.097733 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.097721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:53.097801 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:53.097784 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:53.187512 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.187452 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" event={"ID":"cd749222-a1c6-403b-bcfe-4897707b96cc","Type":"ContainerStarted","Data":"49bbd208237496bb29542483e640f53966a57de81c60d0f7c078dcbc1d41dfb7"} Nov 02 21:33:53.189621 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.189589 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="52515717d0160ba3c28f8085fd2946802ac42a5a017e904e38d3aede15b20815" exitCode=0 Nov 02 21:33:53.189751 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.189697 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerDied","Data":"52515717d0160ba3c28f8085fd2946802ac42a5a017e904e38d3aede15b20815"} Nov 02 21:33:53.300847 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:53.300785 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" podStartSLOduration=3.213345885 podStartE2EDuration="12.300770728s" totalImagesPullingTime="9.087424843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:53.300312052 +0000 UTC m=+12.614571557" watchObservedRunningTime="2026-11-02 21:33:53.300770728 +0000 UTC m=+12.615030274" Nov 02 21:33:54.193135 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:54.193033 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-srw9t" event={"ID":"32d0f122-5c7a-45b3-868e-03157b31421a","Type":"ContainerStarted","Data":"f7d8350fdf5b9119f4096153eb99dde4aa97f35f87f498f9b9b014adb305c04a"} Nov 02 21:33:54.198763 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:54.198701 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-btpw2" event={"ID":"e761f1b3-b683-463e-9c05-6432569d4cd0","Type":"ContainerStarted","Data":"da2946e31be266cb134f1c476ae44bfdbbbe2d9f555236c106f0c3cbddb0de33"} Nov 02 21:33:54.306654 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:54.306595 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-srw9t" podStartSLOduration=2.869348411 podStartE2EDuration="13.306577119s" totalImagesPullingTime="10.437228708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:54.306281957 +0000 UTC m=+13.620541462" watchObservedRunningTime="2026-11-02 21:33:54.306577119 +0000 UTC m=+13.620836625" Nov 02 21:33:55.093800 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:55.093761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:55.093993 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:55.093761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:55.093993 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:55.093883 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:55.093993 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:55.093773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:55.094102 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:55.093989 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:55.094102 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:55.094039 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:55.203563 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:55.203513 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="daccf61c9ea2775b119b2d718f4b43007e65d8dad3cc3f98057045209017376d" exitCode=0 Nov 02 21:33:55.204171 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:55.203592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerDied","Data":"daccf61c9ea2775b119b2d718f4b43007e65d8dad3cc3f98057045209017376d"} Nov 02 21:33:55.318054 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:55.317992 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-btpw2" podStartSLOduration=6.168200645 podStartE2EDuration="14.317972887s" totalImagesPullingTime="8.149772242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:54.420103571 +0000 UTC m=+13.734363087" watchObservedRunningTime="2026-11-02 21:33:55.317972887 +0000 UTC m=+14.632232394" Nov 02 21:33:57.094372 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:57.094339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:57.095046 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:57.094339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:57.095046 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:57.094486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:57.095046 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:57.094447 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:57.095046 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:57.094560 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:57.095046 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:57.094673 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:57.211381 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:57.211348 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="0739a0c8e491fa341ec40a30c4dbc1b9c688f03246eac5a854bea4de70346908" exitCode=0 Nov 02 21:33:57.211547 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:57.211392 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerDied","Data":"0739a0c8e491fa341ec40a30c4dbc1b9c688f03246eac5a854bea4de70346908"} Nov 02 21:33:58.217180 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:58.217136 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"b1599ff5939e13ea74380332897e534a7140d575e1803d40fc400aa6157b483d"} Nov 02 21:33:58.217617 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:58.217278 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"1f5346a4909b513b9da5b87cc84ed2ca39b18a112a1a8823c9d44aaad0440d92"} Nov 02 21:33:58.217617 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:58.217329 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"d2a9d5b33cff38657e9ab3dbcbde92936147a8c4732e9c115b7f459bf8f19139"} Nov 02 21:33:58.878516 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:58.878266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:58.878682 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:58.878544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:58.878682 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:58.878421 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:58.878682 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:58.878653 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:58.878682 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:58.878650 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret podName:befb271e-da36-470a-ae6e-b7ce5dabf394 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.878627375 +0000 UTC m=+34.192886858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret") pod "global-pull-secret-syncer-lmrxw" (UID: "befb271e-da36-470a-ae6e-b7ce5dabf394") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:33:58.878880 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:58.878705 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:14.878688632 +0000 UTC m=+34.192948128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:33:59.079528 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.079500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:59.079673 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.079616 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:33:59.079673 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.079631 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:33:59.079673 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.079640 2580 projected.go:196] Error preparing data for projected volume kube-api-access-j626w for pod openshift-network-diagnostics/network-check-target-gbwsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:59.079790 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.079681 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w podName:92bc027f-95b3-4c18-96b0-9d7bd6ab4413 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:15.079670241 +0000 UTC m=+34.393929724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j626w" (UniqueName: "kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w") pod "network-check-target-gbwsv" (UID: "92bc027f-95b3-4c18-96b0-9d7bd6ab4413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:33:59.093804 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.093772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:33:59.093804 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.093788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:33:59.094019 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.093896 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:33:59.094019 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.093933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:33:59.094019 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.094004 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:33:59.094166 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:33:59.094092 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:33:59.221829 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.221796 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerStarted","Data":"0595796134adad14aa82cc1120e553f000721c06faf93a5970a3d7b6bb878162"} Nov 02 21:33:59.224718 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.224693 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"cf8dc690924e10b8f81a5f29a8959468a5ce3dfb1b5b04cf9df3976f70621bae"} Nov 02 21:33:59.224869 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.224760 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"31919d2a9021c24c6250995fda27f74ed4129d164dccb64b2b2450f1f424fe0f"} Nov 02 21:33:59.224869 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:33:59.224791 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"cccba606154a2faaf17ee8db8fa8b3ea1e9ee78cf409890d305df75575acb78b"} Nov 02 21:34:00.229023 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:00.228987 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="0595796134adad14aa82cc1120e553f000721c06faf93a5970a3d7b6bb878162" exitCode=0 Nov 02 21:34:00.229423 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:00.229040 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerDied","Data":"0595796134adad14aa82cc1120e553f000721c06faf93a5970a3d7b6bb878162"} Nov 02 21:34:01.095005 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:01.094973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:01.095192 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:01.095058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:01.095192 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:01.095098 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:01.095365 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:01.095189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:01.095365 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:01.095217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:01.095365 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:01.095303 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:01.236829 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:01.236789 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"43b468806c827c21d8e60f42b6814531234443a9ed547dafb6f39ba9222a0d14"} Nov 02 21:34:01.239547 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:01.239522 2580 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="cf4bd81ff8afdcfb4f116266582b11a5625a213baa335c08b4fc9164902e201b" exitCode=0 Nov 02 21:34:01.239685 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:01.239553 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerDied","Data":"cf4bd81ff8afdcfb4f116266582b11a5625a213baa335c08b4fc9164902e201b"} Nov 02 21:34:02.244418 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:02.244386 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mv42m" event={"ID":"3a1e59cc-2f43-4079-9713-b19528741a1c","Type":"ContainerStarted","Data":"866202f4d8ec500e876eb5074083c0d2e961ed75689a48766f02ffddb1126c0c"} Nov 02 21:34:02.459180 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:02.459108 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mv42m" podStartSLOduration=9.231137934 podStartE2EDuration="21.45909032s" totalImagesPullingTime="12.227952386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:02.458604605 +0000 UTC m=+21.772864109" watchObservedRunningTime="2026-11-02 21:34:02.45909032 +0000 UTC m=+21.773349824" Nov 02 21:34:03.093665 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.093637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:03.093814 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.093693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:03.093814 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.093704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:03.093814 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:03.093772 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:03.093934 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:03.093876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:03.093934 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:03.093921 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:03.250183 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.250114 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" event={"ID":"4321e24b-79fa-4817-a0eb-66f3d9c875be","Type":"ContainerStarted","Data":"f37a5256a80de4f9b5ed4a65f12b151a69c8e53764364d9ca405f0a8a33c6457"} Nov 02 21:34:03.473868 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.473766 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" podStartSLOduration=8.040526037 podStartE2EDuration="22.473753187s" totalImagesPullingTime="14.43322715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:03.423854569 +0000 UTC m=+22.738114071" watchObservedRunningTime="2026-11-02 21:34:03.473753187 +0000 UTC m=+22.788012716" Nov 02 21:34:03.681523 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.681484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:03.880346 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:03.880266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:04.252373 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:04.252284 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:04.252731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:04.252388 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:04.252731 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:04.252436 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:04.267695 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:04.267670 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:04.267826 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:04.267771 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:04.692240 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:04.692158 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:05.094613 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.094533 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:05.094760 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.094536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:05.094760 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.094667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:05.094760 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:05.094659 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:05.094867 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:05.094756 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:05.094867 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:05.094808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:05.110037 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.110007 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbwsv"] Nov 02 21:34:05.110739 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.110714 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lmrxw"] Nov 02 21:34:05.111528 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.111498 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k75zj"] Nov 02 21:34:05.253830 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.253803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:05.254318 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.253803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:05.254318 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:05.253906 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:05.254318 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:05.253803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:05.254318 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:05.254016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:05.254318 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:05.254057 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:07.094521 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:07.094485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:07.094861 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:07.094494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:07.094861 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:07.094607 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:07.094861 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:07.094485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:07.094861 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:07.094677 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:07.094861 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:07.094760 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:08.531735 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:08.531705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:08.716749 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:08.716713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:08.890916 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:08.890833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:09.055547 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.055513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:09.093573 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.093540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:09.093739 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.093540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:09.093739 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:09.093645 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:09.093739 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.093540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:09.093878 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:09.093742 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:09.093878 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:09.093785 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:09.217332 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.217222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:09.366875 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.366842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:09.524975 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:09.524883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:10.285037 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:10.285007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:10.447755 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:10.447724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:10.858984 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:10.858887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:11.098364 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:11.098324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:11.098570 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:11.098373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:11.098645 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:11.098470 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:11.098974 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:11.098952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmrxw" podUID="befb271e-da36-470a-ae6e-b7ce5dabf394" Nov 02 21:34:11.099095 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:11.099061 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbwsv" podUID="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" Nov 02 21:34:11.099246 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:11.099173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k75zj" podUID="295c327d-c476-4f98-84a1-a39dda118c73" Nov 02 21:34:11.476226 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:11.476034 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeReady" Nov 02 21:34:11.476605 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:11.476327 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:34:11.610429 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:11.610294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:12.412045 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:12.411993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:12.441664 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:12.441639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gkphg_58637e1f-1378-4b0a-9c11-03f3fe45d6ae/dns-node-resolver/0.log" Nov 02 21:34:12.834755 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:12.834675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:12.993575 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:12.993535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:13.097221 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.097120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:13.097221 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.097189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:13.097221 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.097189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:13.149505 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.149481 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:34:13.149798 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.149781 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:34:13.152646 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.152626 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:13.199313 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.199264 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:34:13.199440 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.199422 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zd85x\"" Nov 02 21:34:13.199484 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.199440 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jhcsj\"" Nov 02 21:34:13.199526 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.199494 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fbz7r\"" Nov 02 21:34:13.199577 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.199559 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:34:13.357455 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.357376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:13.443094 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.443069 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wp2q_da3003ef-ed3e-40ea-8dec-781b93a27cc2/node-ca/0.log" Nov 02 21:34:13.506453 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:13.506421 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:14.679193 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:14.679162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:14.894835 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:14.894754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:14.894835 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:14.894795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:14.895037 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:14.895019 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:34:14.895098 ip-10-0-131-47 kubenswrapper[2580]: E1102 21:34:14.895089 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs podName:295c327d-c476-4f98-84a1-a39dda118c73 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:46.895067857 +0000 UTC m=+66.209327340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs") pod "network-metrics-daemon-k75zj" (UID: "295c327d-c476-4f98-84a1-a39dda118c73") : secret "metrics-daemon-secret" not found Nov 02 21:34:14.898537 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:14.898504 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:14.906327 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:14.906306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:14.915336 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:14.915323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:15.080960 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.080924 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:15.095565 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.095531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:15.098714 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.098686 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:15.219980 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.219898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:15.309545 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.309516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:15.411108 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.411078 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lmrxw"] Nov 02 21:34:15.414353 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:34:15.414325 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefb271e_da36_470a_ae6e_b7ce5dabf394.slice/crio-885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5 WatchSource:0}: Error finding container 885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5: Status 404 returned error can't find the container with id 885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5 Nov 02 21:34:15.481684 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.481629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:15.588559 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:15.588498 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbwsv"] Nov 02 21:34:15.592779 ip-10-0-131-47 kubenswrapper[2580]: W1102 21:34:15.592740 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bc027f_95b3_4c18_96b0_9d7bd6ab4413.slice/crio-e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3 WatchSource:0}: Error finding container e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3: Status 404 returned error can't find the container with id e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3 Nov 02 21:34:16.280735 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:16.280703 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbwsv" event={"ID":"92bc027f-95b3-4c18-96b0-9d7bd6ab4413","Type":"ContainerStarted","Data":"e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3"} Nov 02 21:34:16.281729 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:16.281705 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lmrxw" event={"ID":"befb271e-da36-470a-ae6e-b7ce5dabf394","Type":"ContainerStarted","Data":"885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5"} Nov 02 21:34:16.691177 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:16.691079 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:16.890180 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:16.890109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:17.089697 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:17.089615 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:17.282040 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:17.281995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:17.488652 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:17.488572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:17.638202 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:17.638163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:18.683885 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:18.683850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:18.883365 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:18.883303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:19.087800 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:19.087681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:19.292631 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:19.292583 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:19.481665 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:19.481578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:20.063036 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.062990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:20.293855 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.293817 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbwsv" event={"ID":"92bc027f-95b3-4c18-96b0-9d7bd6ab4413","Type":"ContainerStarted","Data":"5b2f798f0e89b376d773f137cf02a30375bec383f7bf21902b31b890f7c6cb8b"} Nov 02 21:34:20.294029 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.293943 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:20.324072 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.324005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:20.410361 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.410301 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbwsv" podStartSLOduration=35.472155405 podStartE2EDuration="39.410282578s" totalImagesPullingTime="3.938127173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:20.409713118 +0000 UTC m=+39.723972621" watchObservedRunningTime="2026-11-02 21:34:20.410282578 +0000 UTC m=+39.724542082" Nov 02 21:34:20.477386 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.477353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:20.683025 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.683003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:20.839054 ip-10-0-131-47 kubenswrapper[2580]: I1102 21:34:20.838988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:20.977894 ip-10-0-131-47 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:34:20.997485 ip-10-0-131-47 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:34:20.997651 ip-10-0-131-47 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:34:20.997698 ip-10-0-131-47 systemd[1]: kubelet.service: Consumed 2.261s CPU time. Nov 02 21:34:21.007788 ip-10-0-131-47 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:34:21.095745 ip-10-0-131-47 kubenswrapper[4285]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:34:21.095745 ip-10-0-131-47 kubenswrapper[4285]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:34:21.095745 ip-10-0-131-47 kubenswrapper[4285]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:34:21.095745 ip-10-0-131-47 kubenswrapper[4285]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:34:21.097830 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097748 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.097830 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097829 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.097830 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097833 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097837 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097840 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097845 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097847 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097850 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097853 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097856 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097858 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097861 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097863 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097866 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097869 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097872 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097874 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097877 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097880 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097883 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097885 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097888 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.097928 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097891 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097893 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097896 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097902 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097905 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097908 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097911 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097914 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097916 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097919 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097922 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097924 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097927 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097930 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097933 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097935 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097938 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097941 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097944 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.098429 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097947 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097950 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097953 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097955 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097958 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097960 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097962 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097965 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097968 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097970 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097973 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097975 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097978 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097980 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097983 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097988 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097992 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097995 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.097998 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098001 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.098892 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098005 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098010 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098013 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098015 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098018 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098021 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098023 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098026 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098029 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098031 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098034 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098036 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098039 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098042 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098044 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098047 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098049 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098052 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098055 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.099385 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098057 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098060 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098063 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098065 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098067 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098070 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098073 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098075 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098078 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098080 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098083 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098086 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098089 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098092 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098095 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098097 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098100 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098102 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098105 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098108 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.099855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098110 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098113 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098117 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098119 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098137 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098142 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098933 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098941 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098945 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098949 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098952 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098955 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098957 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098960 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098963 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098966 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098968 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098972 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098974 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.100390 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098977 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098979 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098982 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098985 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098987 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.098998 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099001 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099003 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099006 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099010 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099015 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099018 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099021 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099023 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099026 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099029 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099031 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099034 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099037 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099039 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.100855 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099042 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099045 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099047 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099050 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099053 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099055 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099058 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099060 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099062 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099065 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099068 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099070 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099073 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099075 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099077 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099081 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099084 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099087 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099090 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099093 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.101391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099096 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099098 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099101 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099103 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099106 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099109 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099111 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099113 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099116 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099118 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099137 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099141 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099145 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099148 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099151 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099153 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099156 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099160 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099162 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099165 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.101875 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099168 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099170 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099173 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099176 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099179 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099182 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099184 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099187 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099190 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099192 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099195 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099199 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099201 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099204 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099207 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099209 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099213 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099217 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099221 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.102413 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099224 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099227 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099230 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099233 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099236 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099238 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099241 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099244 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099246 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099249 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099251 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099254 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099257 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099260 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099777 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099782 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099785 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099788 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099791 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099794 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.102882 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099796 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099799 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099801 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099804 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099808 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099810 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099813 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099816 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099818 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099821 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099824 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099827 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099829 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099832 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099835 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099838 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099840 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099843 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099846 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.103391 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099849 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099851 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099855 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099857 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099860 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099863 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099866 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099868 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099871 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099873 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099876 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099878 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099881 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099883 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099886 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099888 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099891 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099893 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099897 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.103864 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099901 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099903 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099906 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099909 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099911 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099914 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099917 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099920 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099923 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099925 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099928 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099930 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099932 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099935 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099938 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099941 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099944 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099946 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099949 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099952 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.104493 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099954 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099957 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099959 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099963 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099965 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099967 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099970 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099972 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099975 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099978 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099981 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099983 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099987 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099990 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099992 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099995 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.099997 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100000 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100002 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100006 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.105332 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100008 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100011 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100014 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100016 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100019 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100021 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100024 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100028 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100032 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100035 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100039 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100041 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100045 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100047 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100050 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100052 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100055 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100059 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100061 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.105816 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100063 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100066 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100069 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100175 4285 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100183 4285 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100189 4285 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100195 4285 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100200 4285 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100203 4285 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100207 4285 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100211 4285 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100215 4285 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100218 4285 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100221 4285 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100225 4285 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100228 4285 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100231 4285 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100234 4285 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100237 4285 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100240 4285 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100243 4285 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100248 4285 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100251 4285 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:34:21.106410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100254 4285 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100257 4285 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100261 4285 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100265 4285 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100268 4285 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100271 4285 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100275 4285 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100278 4285 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100281 4285 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100284 4285 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100287 4285 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100290 4285 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100295 4285 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100298 4285 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100301 4285 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100304 4285 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100308 4285 flags.go:64] FLAG: --enable-server="true" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100310 4285 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100314 4285 flags.go:64] FLAG: --event-burst="100" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100317 4285 flags.go:64] FLAG: --event-qps="50" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100320 4285 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100323 4285 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100326 4285 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100333 4285 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:34:21.106964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100336 4285 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100339 4285 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100342 4285 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100345 4285 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100348 4285 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100351 4285 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100355 4285 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100358 4285 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100361 4285 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100364 4285 flags.go:64] FLAG: --feature-gates="" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100368 4285 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100372 4285 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100375 4285 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100378 4285 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100381 4285 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100384 4285 flags.go:64] FLAG: --help="false" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100387 4285 flags.go:64] FLAG: --hostname-override="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100391 4285 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100394 4285 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100397 4285 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100400 4285 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100404 4285 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100407 4285 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:34:21.107553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100410 4285 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100413 4285 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100417 4285 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100420 4285 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100423 4285 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100426 4285 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100429 4285 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100432 4285 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100436 4285 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100439 4285 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100442 4285 flags.go:64] FLAG: --lock-file="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100445 4285 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100448 4285 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100451 4285 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100456 4285 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100459 4285 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100462 4285 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100465 4285 flags.go:64] FLAG: --logging-format="text" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100468 4285 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100472 4285 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100475 4285 flags.go:64] FLAG: --manifest-url="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100478 4285 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100482 4285 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100485 4285 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100490 4285 flags.go:64] FLAG: --max-pods="110" Nov 02 21:34:21.108103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100493 4285 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100497 4285 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100500 4285 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100503 4285 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100505 4285 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100508 4285 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100511 4285 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100518 4285 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100522 4285 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100525 4285 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100529 4285 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100532 4285 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100535 4285 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100538 4285 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100541 4285 flags.go:64] FLAG: --port="10250" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100545 4285 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100548 4285 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-094e21b90004baa96" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100551 4285 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100555 4285 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100558 4285 flags.go:64] FLAG: --register-node="true" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100561 4285 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100565 4285 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100568 4285 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100570 4285 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100573 4285 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100577 4285 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:34:21.108720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100581 4285 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100584 4285 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100587 4285 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100590 4285 flags.go:64] FLAG: --runonce="false" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100593 4285 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100596 4285 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100599 4285 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100602 4285 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100605 4285 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100609 4285 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100612 4285 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100615 4285 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100618 4285 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100621 4285 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100624 4285 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100627 4285 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100630 4285 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100633 4285 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100636 4285 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100641 4285 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100644 4285 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100649 4285 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100653 4285 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100657 4285 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100659 4285 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:34:21.109358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100663 4285 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100666 4285 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100669 4285 flags.go:64] FLAG: --v="2" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100673 4285 flags.go:64] FLAG: --version="false" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100678 4285 flags.go:64] FLAG: --vmodule="" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100682 4285 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.100685 4285 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100830 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100834 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100837 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100839 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100844 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100848 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100852 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100855 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100858 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100860 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100863 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100866 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100869 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.109963 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100872 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100874 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100877 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100879 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100882 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100885 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100887 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100890 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100894 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100897 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100899 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100902 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100905 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100907 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100910 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100913 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100915 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100918 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100921 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.110460 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100923 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100927 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100930 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100933 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100935 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100938 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100940 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100943 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100945 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100948 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100950 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100953 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100956 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100958 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100961 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100964 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100966 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100969 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100971 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100974 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.110921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100977 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100981 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100983 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100986 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100988 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100991 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100994 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100996 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.100999 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101002 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101004 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101007 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101009 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101013 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101015 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101018 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101020 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101023 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101026 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.111676 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101029 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101031 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101034 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101037 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101039 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101042 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101044 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101047 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101049 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101052 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101054 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101057 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101059 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101062 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101066 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101069 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101072 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101074 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101077 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101080 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.112231 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101082 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101085 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101089 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101093 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101096 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101098 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101102 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101105 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101107 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101110 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101113 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101115 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101118 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101134 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101138 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.112878 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.101172 4285 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101213 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101219 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101222 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101225 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101227 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101230 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101232 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101235 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101238 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101242 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101250 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101255 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101258 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101261 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101263 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101266 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101268 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101271 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.113360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101274 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101276 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101279 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101281 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101285 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101288 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101291 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101294 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101296 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101299 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101302 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101304 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101307 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101309 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101312 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101314 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101317 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101320 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101322 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.113817 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101325 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101328 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101330 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101333 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101335 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101338 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101343 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101346 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101348 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101351 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101353 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101356 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101358 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101361 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101363 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101366 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101368 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101372 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101375 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101377 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.114403 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101380 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101383 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101385 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101388 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101390 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101393 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101396 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101399 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101401 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101404 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101407 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101409 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101412 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101414 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101417 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101419 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101422 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101425 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101430 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.115087 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101433 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101435 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101438 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101441 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101443 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101446 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101448 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101451 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101453 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101456 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101460 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101462 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101465 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101468 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101470 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101473 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101475 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101478 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101480 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101483 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.115672 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101486 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101488 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101492 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101496 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101499 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101502 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101504 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101506 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101509 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.101511 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.106535 4285 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.106551 4285 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.106575 4285 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.106580 4285 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106639 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106644 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106647 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106650 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106654 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.116172 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106656 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106659 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106662 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106665 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106668 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106672 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106676 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106679 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106682 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106685 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106687 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106690 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106693 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106696 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106698 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106701 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106704 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106706 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106709 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106712 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.116632 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106714 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106717 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106720 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106722 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106725 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106728 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106730 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106733 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106736 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106738 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106741 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106744 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106746 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106749 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106751 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106754 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106757 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106759 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106762 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106765 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.117114 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106769 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106773 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106777 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106780 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106783 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106786 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106789 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106791 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106794 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106797 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106799 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106802 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106805 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106808 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106810 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106813 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106816 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106818 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106821 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.117697 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106824 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106827 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106829 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106832 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106835 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106837 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106840 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106842 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106845 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106848 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106850 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106853 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106855 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106858 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106860 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106863 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106866 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106869 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106872 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.118163 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106874 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106877 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106879 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106882 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106884 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106887 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106889 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106892 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106895 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106898 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106900 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106903 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106906 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106908 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106911 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106913 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106916 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106918 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106921 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106923 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.118616 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106926 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106928 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.106931 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.106956 4285 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107103 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107107 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107111 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107114 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107117 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107136 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107140 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107142 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107145 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107148 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107151 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107154 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.119105 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107157 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107160 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107163 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107165 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107168 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107170 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107173 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107175 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107178 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107181 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107183 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107186 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107188 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107191 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107194 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107196 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107199 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107201 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107204 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107206 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.119503 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107210 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107212 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107215 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107218 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107220 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107223 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107227 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107230 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107233 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107235 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107238 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107240 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107243 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107245 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107248 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107251 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107253 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107257 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107261 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.120091 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107265 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107268 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107271 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107273 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107276 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107279 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107281 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107284 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107286 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107289 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107292 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107295 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107298 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107301 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107304 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107306 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107309 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107312 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107314 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.120850 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107317 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107320 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107323 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107326 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107328 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107331 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107333 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107336 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107338 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107341 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107344 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107346 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107348 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107351 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107354 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107356 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107358 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107361 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107363 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.121459 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107366 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107368 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107371 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107374 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107376 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107379 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107382 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107387 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107389 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107392 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107395 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107397 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107400 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107403 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107405 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107409 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.121913 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107412 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.107436 4285 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107467 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107472 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107475 4285 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107478 4285 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107480 4285 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107483 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107486 4285 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107488 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107491 4285 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107494 4285 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107496 4285 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107499 4285 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107501 4285 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107506 4285 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:34:21.122360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107509 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107512 4285 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107514 4285 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107517 4285 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107519 4285 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107522 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107525 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107527 4285 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107531 4285 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107534 4285 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107537 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107539 4285 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107542 4285 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107544 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107547 4285 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107549 4285 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107552 4285 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107555 4285 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107558 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107561 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:34:21.122749 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107564 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107567 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107570 4285 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107572 4285 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107575 4285 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107578 4285 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107580 4285 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107583 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107585 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107588 4285 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107591 4285 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107595 4285 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107598 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107600 4285 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107603 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107605 4285 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107608 4285 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107610 4285 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107613 4285 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107615 4285 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:34:21.123250 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107620 4285 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107622 4285 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107625 4285 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107629 4285 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107633 4285 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107636 4285 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107639 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107642 4285 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107645 4285 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107647 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107650 4285 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107653 4285 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107656 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107658 4285 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107661 4285 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107664 4285 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107668 4285 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107671 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107673 4285 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:34:21.123769 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107676 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107679 4285 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107681 4285 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107684 4285 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107688 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107690 4285 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107693 4285 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107695 4285 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107698 4285 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107701 4285 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107703 4285 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107705 4285 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107708 4285 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107712 4285 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107714 4285 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107717 4285 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107720 4285 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107722 4285 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107727 4285 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107730 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:34:21.124240 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107732 4285 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107735 4285 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107737 4285 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107740 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107743 4285 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107745 4285 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107748 4285 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107750 4285 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107753 4285 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107756 4285 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107758 4285 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107761 4285 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:21.107764 4285 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.107817 4285 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.108850 4285 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.108903 4285 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.109085 4285 server.go:1014] "Starting client certificate rotation" Nov 02 21:34:21.125049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.109237 4285 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.109278 4285 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:28:41 +0000 UTC" deadline="2028-05-17 11:11:54.121915647 +0000 UTC" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.109305 4285 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13477h37m33.012612524s" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.109573 4285 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.110609 4285 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.112304 4285 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.114007 4285 log.go:25] "Validated CRI v1 image API" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.114810 4285 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:34:21.125796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.116329 4285 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a7ab3d2a-a85d-47cf-b1d9-e7c412a8210e:/dev/nvme0n1p3 d088ad82-2b06-4530-becd-c3a69f58ebf6:/dev/nvme0n1p4] Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.116348 4285 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2615d298-b340-465b-b9b0-f66aa0f59628/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2615d298-b340-465b-b9b0-f66aa0f59628/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/295c327d-c476-4f98-84a1-a39dda118c73/volumes/kubernetes.io~projected/kube-api-access-fddrs:{mountpoint:/var/lib/kubelet/pods/295c327d-c476-4f98-84a1-a39dda118c73/volumes/kubernetes.io~projected/kube-api-access-fddrs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~projected/kube-api-access-5t8d2:{mountpoint:/var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~projected/kube-api-access-5t8d2 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a1e59cc-2f43-4079-9713-b19528741a1c/volumes/kubernetes.io~projected/kube-api-access-hv79b:{mountpoint:/var/lib/kubelet/pods/3a1e59cc-2f43-4079-9713-b19528741a1c/volumes/kubernetes.io~projected/kube-api-access-hv79b major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volumes/kubernetes.io~projected/kube-api-access-rzf29:{mountpoint:/var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volumes/kubernetes.io~projected/kube-api-access-rzf29 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58637e1f-1378-4b0a-9c11-03f3fe45d6ae/volumes/kubernetes.io~projected/kube-api-access-mlgf2:{mountpoint:/var/lib/kubelet/pods/58637e1f-1378-4b0a-9c11-03f3fe45d6ae/volumes/kubernetes.io~projected/kube-api-access-mlgf2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92bc027f-95b3-4c18-96b0-9d7bd6ab4413/volumes/kubernetes.io~projected/kube-api-access-j626w:{mountpoint:/var/lib/kubelet/pods/92bc027f-95b3-4c18-96b0-9d7bd6ab4413/volumes/kubernetes.io~projected/kube-api-access-j626w major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befb271e-da36-470a-ae6e-b7ce5dabf394/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/befb271e-da36-470a-ae6e-b7ce5dabf394/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd749222-a1c6-403b-bcfe-4897707b96cc/volumes/kubernetes.io~projected/kube-api-access-zlvqd:{mountpoint:/var/lib/kubelet/pods/cd749222-a1c6-403b-bcfe-4897707b96cc/volumes/kubernetes.io~projected/kube-api-access-zlvqd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfd48ab2-58a6-42c2-8077-3b9af1669069/volumes/kubernetes.io~projected/kube-api-access-fq4tv:{mountpoint:/var/lib/kubelet/pods/cfd48ab2-58a6-42c2-8077-3b9af1669069/volumes/kubernetes.io~projected/kube-api-access-fq4tv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3003ef-ed3e-40ea-8dec-781b93a27cc2/volumes/kubernetes.io~projected/kube-api-access-b8sjh:{mountpoint:/var/lib/kubelet/pods/da3003ef-ed3e-40ea-8dec-781b93a27cc2/volumes/kubernetes.io~projected/kube-api-access-b8sjh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e761f1b3-b683-463e-9c05-6432569d4cd0/volumes/kubernetes.io~projected/kube-api-access-n46x6:{mountpoint:/var/lib/kubelet/pods/e761f1b3-b683-463e-9c05-6432569d4cd0/volumes/kubernetes.io~projected/kube-api-access-n46x6 major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/294b927a297e5f997b15d56b821c82c5dbc27f96fdb42bd962b0b6cf0c08a0dd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0209bd4389152cc6bc2db6cb8bd9a07b0aa49cbe40beb5d3e69c5e595e7bc92d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/3dea6a8572ce26da04d736251b6c3ce1db001381f60478f7bd32593ab42a461d/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/983657c149ff257799149d3c709a7a35fd730fcdf980dc37ba92bc01ceda65b7/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1fb2a8a334830f761436f086e33e89786cd42dbca4d4694a2c6db8084f3579c2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8628c172cfe4980cc2556e10c82c24b198311d36b49aeb883334f13845919f19/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/36c6ca80bc6468844cd4fd477c8f211fcfef10ef2d391de9e37296e772862d48/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/69ab3311d8128100a86706e6b2bfea554b451fe0ff0eea92a2d29f9c10478be8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/18bdf1b74f5a2b61b459b1ff85f528a89e7e36892d9f6d4b7c9cd223ff7d9c98/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/59d3f055f070d6d55340a8d42a511250f5d58f6beb6fffbc492c67effcdf4134/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/daff9844c28863f0bc9f481c825952f5c7dab3a91e61afb3e10e6cdb8fa7dd5f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8d7851af65af3ab786618660aaa230d8e2b853688f3e79431bbad5dddf4bf12b/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6075d13a698db9e431098f5f42b4799f78910cf69b2756841fd15d06f77c30c3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2ac5231329d965be54626b9d103ba90284dafeea85067dfa3b7c257c62112f37/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3e45c842ebe29b36a8cfb67dab5da98f243ce8bdafb10986c49448f973f7bd6e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fc5d17b405efd34b5aa2e8574487403d05c2ac01397adeb7cbb8a1590e51dfb3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/82289dd328003ca9974f00c168fe0e5759c422f1b0a2fba025451630dbb7611a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5c414fadd0e539ba55dbe4c359b8db697cf726986e6627bc1aab6b3924f6deb2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e00e3019d4f8eb23db2a81edc3ca36c8a2a543a0b6fa0cb820db283ee16e78dd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3b5ceb36f05fdea26e4aa3bd7f980c23329e771695ba9009989c44381745ec2d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6775a40dda2dcb7e4d954c6e7a0740645082454e1a896ec55391482e1c4455e2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/cbcde757f1a9ded94688a182805e0bcbe22a5e44d84d39cc2d663fee861c8dd5/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/a0de2bfceba406634cc0d84fdb78dcef51f5be456a3f24e3c6aa364b4adc989f/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/be8bc09085ac5b2ecb71943662d903c1dfcb9dbcf50df485f03329c3a4a40a76/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/6fae36a05391a2d507cf37cd3769d492fc8539c8f2f289f8c0aee81d887efe73/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/edb8405f4f4f8acce7fd61f93fc94d37dc5596b73f2d170b0dd1edacd390bb91/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/72c6f58dcb88156613fed3aad8126efbb331940c1cff4dbe6b5b183ecda0f8bb/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7917e7389731d2b36c5106e8e6620fa0972c68cd9f4301f12e38680cf2221f72/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d96e55364fb26ddf556f41241782e0e0e488861c01e0478731572c6bc099e160/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/cd57fc353f296fc541515ca784235211e08c28a896fa17e40b72dfbb28055a96/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1bb24d7d7bc86ddc43e78d2b94a7108b9b3ef7934c36f8ade399a3ef2f94c3fc/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c81c2145b0e4cb250e011555e7e296548574b1b385816c965cde2cc770634c12/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ba4d8499929bcf23374dde2d1fc9aad025bcdfb00c5d5eed07e5a2dea74a5b06/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4a332afa0de678257915a1b675acc42afcb3d8d0fe2c36a6bc4ee325bb1b58aa/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/06dc516f4cfe2f226583d7bc5a15e014e5b979c59b08d9247aa9bbc717da4bea/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123121 4285 manager.go:217] Machine: {Timestamp:2026-11-02 21:34:21.122630678 +0000 UTC m=+0.084180988 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127216 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229189611e0c7efb84ea60d137be64 SystemUUID:ec229189-611e-0c7e-fb84-ea60d137be64 BootID:34d7e761-90cb-433c-9d1e-6431e3cb858c Filesystems:[{Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfd48ab2-58a6-42c2-8077-3b9af1669069/volumes/kubernetes.io~projected/kube-api-access-fq4tv DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f32202c8e42a097fcff7a664e1412c66d7c82e57fc5a330d8df145b821a92422/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/befb271e-da36-470a-ae6e-b7ce5dabf394/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/843c16a0eb0a15fc6544834bd7603519305f4c6b22886b5389bec157bda84ce4/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0bd466402ef1214fa8f365ec516c892e95b72e1f39ff455f39d0a2a0ce976f3/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~projected/kube-api-access-5t8d2 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5673a1b6b0c397b036e1dc4dcdd35d061049e5080e71608257075cc1b2fba3c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e8a18787a656515d253d4cf670224374011e9310617b34558926fde26285984/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a1e59cc-2f43-4079-9713-b19528741a1c/volumes/kubernetes.io~projected/kube-api-access-hv79b DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8f9c86b6d535c8908fffda042b7d526ca45a4f30a4dc2149a15ceca5256af8a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/885095dbb456c34130d0d2cae8e971889a650458d236475babce5bd7b15850a5/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/32d0f122-5c7a-45b3-868e-03157b31421a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volumes/kubernetes.io~projected/kube-api-access-rzf29 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58637e1f-1378-4b0a-9c11-03f3fe45d6ae/volumes/kubernetes.io~projected/kube-api-access-mlgf2 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/295c327d-c476-4f98-84a1-a39dda118c73/volumes/kubernetes.io~projected/kube-api-access-fddrs DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e761f1b3-b683-463e-9c05-6432569d4cd0/volumes/kubernetes.io~projected/kube-api-access-n46x6 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd749222-a1c6-403b-bcfe-4897707b96cc/volumes/kubernetes.io~projected/kube-api-access-zlvqd DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4321e24b-79fa-4817-a0eb-66f3d9c875be/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db95c1d0a12474082ae7fcf663db51cc8a2846b4b9df3f2d338b2a4005ed22a0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2615d298-b340-465b-b9b0-f66aa0f59628/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da3003ef-ed3e-40ea-8dec-781b93a27cc2/volumes/kubernetes.io~projected/kube-api-access-b8sjh DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24bbef1ec2978309d7ab3524394a0421566541c8f59727287a0e7760b7d4011b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92bc027f-95b3-4c18-96b0-9d7bd6ab4413/volumes/kubernetes.io~projected/kube-api-access-j626w DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b3cf92226df7f7ede0bba39447d35c86ff49375532a3aacc086214537b79d2d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:885095dbb456c34 MacAddress:62:d8:41:ef:0e:b3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:69:af:82:d2:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:ca:67:19:48:db:82 Speed:0 Mtu:8901} {Name:e0bd466402ef121 MacAddress:66:c5:90:c9:82:ce Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:69:af:82:d2:3f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2e:08:65:b3:eb:e3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:67:91:54:49:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123375 4285 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123418 4285 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123666 4285 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123696 4285 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-47.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123820 4285 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123828 4285 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123840 4285 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123853 4285 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.123933 4285 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124090 4285 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124210 4285 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124243 4285 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:34:21.126188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124252 4285 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124262 4285 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124270 4285 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124279 4285 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124805 4285 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.124814 4285 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.125279 4285 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.126983 4285 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.126993 4285 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:34:21.127189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127105 4285 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127304 4285 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127332 4285 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127422 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127438 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127444 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127449 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127455 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127461 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127468 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127474 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127482 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127490 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127507 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127518 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127546 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:34:21.127575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.127552 4285 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:34:21.130859 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.130843 4285 server.go:1263] "Started kubelet" Nov 02 21:34:21.130957 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.130911 4285 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:34:21.131550 ip-10-0-131-47 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:34:21.132355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.132013 4285 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:34:21.132355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.132292 4285 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:34:21.133232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.133182 4285 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:34:21.133232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.133228 4285 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.135168 4285 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.135835 4285 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.135911 4285 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.136062 4285 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.136068 4285 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.136108 4285 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.136162 4285 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:28:43 +0000 UTC" deadline="2028-08-13 08:20:27.1088959 +0000 UTC" Nov 02 21:34:21.137557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.136195 4285 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15586h46m5.97270438s" Nov 02 21:34:21.138026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.137902 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e24b-79fa-4817-a0eb-66f3d9c875be" volumeName="kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-config" seLinuxMountContext="" Nov 02 21:34:21.138026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.137970 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e24b-79fa-4817-a0eb-66f3d9c875be" volumeName="kubernetes.io/projected/4321e24b-79fa-4817-a0eb-66f3d9c875be-kube-api-access-rzf29" seLinuxMountContext="" Nov 02 21:34:21.138026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.137985 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfd48ab2-58a6-42c2-8077-3b9af1669069" volumeName="kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-daemon-config" seLinuxMountContext="" Nov 02 21:34:21.138026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138002 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfd48ab2-58a6-42c2-8077-3b9af1669069" volumeName="kubernetes.io/projected/cfd48ab2-58a6-42c2-8077-3b9af1669069-kube-api-access-fq4tv" seLinuxMountContext="" Nov 02 21:34:21.138026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138024 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e761f1b3-b683-463e-9c05-6432569d4cd0" volumeName="kubernetes.io/projected/e761f1b3-b683-463e-9c05-6432569d4cd0-kube-api-access-n46x6" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138038 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1e59cc-2f43-4079-9713-b19528741a1c" volumeName="kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-binary-copy" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138056 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1e59cc-2f43-4079-9713-b19528741a1c" volumeName="kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138077 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e24b-79fa-4817-a0eb-66f3d9c875be" volumeName="kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138100 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e24b-79fa-4817-a0eb-66f3d9c875be" volumeName="kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-env-overrides" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138120 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58637e1f-1378-4b0a-9c11-03f3fe45d6ae" volumeName="kubernetes.io/projected/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-kube-api-access-mlgf2" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138157 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfd48ab2-58a6-42c2-8077-3b9af1669069" volumeName="kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-cni-binary-copy" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138174 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2615d298-b340-465b-b9b0-f66aa0f59628" volumeName="kubernetes.io/configmap/2615d298-b340-465b-b9b0-f66aa0f59628-konnectivity-ca" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138194 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="295c327d-c476-4f98-84a1-a39dda118c73" volumeName="kubernetes.io/projected/295c327d-c476-4f98-84a1-a39dda118c73-kube-api-access-fddrs" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138220 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d0f122-5c7a-45b3-868e-03157b31421a" volumeName="kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-tmp" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138239 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1e59cc-2f43-4079-9713-b19528741a1c" volumeName="kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:34:21.138277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138255 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58637e1f-1378-4b0a-9c11-03f3fe45d6ae" volumeName="kubernetes.io/empty-dir/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-tmp-dir" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138276 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92bc027f-95b3-4c18-96b0-9d7bd6ab4413" volumeName="kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138309 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd749222-a1c6-403b-bcfe-4897707b96cc" volumeName="kubernetes.io/projected/cd749222-a1c6-403b-bcfe-4897707b96cc-kube-api-access-zlvqd" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138324 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3003ef-ed3e-40ea-8dec-781b93a27cc2" volumeName="kubernetes.io/configmap/da3003ef-ed3e-40ea-8dec-781b93a27cc2-serviceca" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138346 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2615d298-b340-465b-b9b0-f66aa0f59628" volumeName="kubernetes.io/secret/2615d298-b340-465b-b9b0-f66aa0f59628-agent-certs" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138365 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d0f122-5c7a-45b3-868e-03157b31421a" volumeName="kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-etc-tuned" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138394 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d0f122-5c7a-45b3-868e-03157b31421a" volumeName="kubernetes.io/projected/32d0f122-5c7a-45b3-868e-03157b31421a-kube-api-access-5t8d2" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138461 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a1e59cc-2f43-4079-9713-b19528741a1c" volumeName="kubernetes.io/projected/3a1e59cc-2f43-4079-9713-b19528741a1c-kube-api-access-hv79b" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138485 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4321e24b-79fa-4817-a0eb-66f3d9c875be" volumeName="kubernetes.io/secret/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138499 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befb271e-da36-470a-ae6e-b7ce5dabf394" volumeName="kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138519 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3003ef-ed3e-40ea-8dec-781b93a27cc2" volumeName="kubernetes.io/projected/da3003ef-ed3e-40ea-8dec-781b93a27cc2-kube-api-access-b8sjh" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138536 4285 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e761f1b3-b683-463e-9c05-6432569d4cd0" volumeName="kubernetes.io/configmap/e761f1b3-b683-463e-9c05-6432569d4cd0-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138548 4285 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:34:21.138738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.138555 4285 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:34:21.140454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140437 4285 factory.go:55] Registering systemd factory Nov 02 21:34:21.140524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140458 4285 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:34:21.140782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140760 4285 factory.go:153] Registering CRI-O factory Nov 02 21:34:21.140782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140786 4285 factory.go:223] Registration of the crio container factory successfully Nov 02 21:34:21.140902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140838 4285 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:34:21.140902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140854 4285 factory.go:103] Registering Raw factory Nov 02 21:34:21.140902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.140867 4285 manager.go:1199] Started watching for new ooms in manager Nov 02 21:34:21.141188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.141178 4285 manager.go:319] Starting recovery of all containers Nov 02 21:34:21.165149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.162115 4285 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:34:21.165413 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.165396 4285 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:34:21.165615 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.165594 4285 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:34:21.165720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.165623 4285 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:34:21.165720 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.165690 4285 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:34:21.174597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.174558 4285 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="cf4bd81ff8afdcfb4f116266582b11a5625a213baa335c08b4fc9164902e201b" exitCode=0 Nov 02 21:34:21.174597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.174593 4285 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="0595796134adad14aa82cc1120e553f000721c06faf93a5970a3d7b6bb878162" exitCode=0 Nov 02 21:34:21.174597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.174599 4285 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="0739a0c8e491fa341ec40a30c4dbc1b9c688f03246eac5a854bea4de70346908" exitCode=0 Nov 02 21:34:21.174785 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.174607 4285 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="daccf61c9ea2775b119b2d718f4b43007e65d8dad3cc3f98057045209017376d" exitCode=0 Nov 02 21:34:21.174785 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.174612 4285 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="52515717d0160ba3c28f8085fd2946802ac42a5a017e904e38d3aede15b20815" exitCode=0 Nov 02 21:34:21.174785 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.174618 4285 generic.go:356] "Generic (PLEG): container finished" podID="3a1e59cc-2f43-4079-9713-b19528741a1c" containerID="89e060b6c17124ef2e02d179f3baf035d789dfdd08dc3f55416141c55354d52f" exitCode=0 Nov 02 21:34:21.193197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.193170 4285 generic.go:356] "Generic (PLEG): container finished" podID="c9d7a9d6fa496da9c68538ee1b34b8e7" containerID="253d60cdea4799fe449cf3f3231013836cc6d0da0150054c9ff960b3aa52211d" exitCode=0 Nov 02 21:34:21.199694 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.199677 4285 manager.go:324] Recovery completed Nov 02 21:34:21.212957 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.212936 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.213948 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.213931 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.214028 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.213960 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.214028 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.213970 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.215171 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215153 4285 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:34:21.215171 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215167 4285 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:34:21.215310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215179 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.215310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215186 4285 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:34:21.215406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215335 4285 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:34:21.215406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215351 4285 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:34:21.215406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215373 4285 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:34:21.215406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215382 4285 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:34:21.215637 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215426 4285 policy_none.go:50] "Start" Nov 02 21:34:21.215637 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215439 4285 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:34:21.215637 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215453 4285 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:34:21.215778 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215663 4285 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:34:21.215778 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215675 4285 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:34:21.215778 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215684 4285 policy_none.go:44] "Start" Nov 02 21:34:21.215953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215938 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.216005 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215965 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.216005 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.215979 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.219618 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.219586 4285 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:34:21.222738 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.222726 4285 manager.go:341] "Starting Device Plugin manager" Nov 02 21:34:21.222866 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.222748 4285 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:34:21.222866 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.222757 4285 server.go:87] "Starting device plugin registration server" Nov 02 21:34:21.223000 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.222989 4285 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:34:21.223042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.223001 4285 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:34:21.223663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.223114 4285 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:34:21.223663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.223216 4285 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:34:21.223663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.223249 4285 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:34:21.223663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.223256 4285 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:34:21.223663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.223304 4285 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:34:21.22324068Z","UUID":"42453ed2-73ca-42e2-b242-9a5755a00167","Handler":null,"Name":"","Endpoint":""} Nov 02 21:34:21.224541 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.224515 4285 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:34:21.224627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.224531 4285 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:34:21.224627 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.224611 4285 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" Nov 02 21:34:21.224713 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.224635 4285 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:34:21.266670 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.266184 4285 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal"] Nov 02 21:34:21.266793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.266718 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.267620 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.267605 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.267696 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.267633 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.267696 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.267648 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.267865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.267854 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.268007 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.267995 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.268626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268601 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.268724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268639 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.268724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268605 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.268724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268675 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.268724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268687 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.268724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268656 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.268871 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.268852 4285 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.268938 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268910 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"c9d7a9d6fa496da9c68538ee1b34b8e7","Type":"ContainerStarted","Data":"b713314f30527f151e8bb45d8bc2caf1c6c511aaaee524522e28b8f1ece6b211"} Nov 02 21:34:21.268970 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268945 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"c9d7a9d6fa496da9c68538ee1b34b8e7","Type":"ContainerDied","Data":"253d60cdea4799fe449cf3f3231013836cc6d0da0150054c9ff960b3aa52211d"} Nov 02 21:34:21.268998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268968 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"c9d7a9d6fa496da9c68538ee1b34b8e7","Type":"ContainerStarted","Data":"7f7c59b3d24fb437ab36813ce990e43331af1d9e5c14549520aa244c406985df"} Nov 02 21:34:21.268998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268980 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"ef8aa625af233e1fa89cb16cb4aca78e","Type":"ContainerStarted","Data":"d6efb358e248d309b0195e01ff9e88e85613f8456828a39bb1c0a3404934f107"} Nov 02 21:34:21.268998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268983 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.268998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.268992 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" event={"ID":"ef8aa625af233e1fa89cb16cb4aca78e","Type":"ContainerStarted","Data":"64f75fcc2747320b096463e54f090c47e2b5bd41ad1b7d8cc809d4732d15f935"} Nov 02 21:34:21.269718 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.269704 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.269778 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.269733 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.269778 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.269748 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.269885 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.269874 4285 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-47.us-west-1.compute.internal\" not found" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.323397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.323373 4285 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:34:21.324298 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.324283 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:34:21.324365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.324312 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:34:21.324365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.324327 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 21:34:21.324365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.324354 4285 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.332986 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.332967 4285 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:34:21.333045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.333032 4285 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:21.381296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.381244 4285 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:21.381296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.381248 4285 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:21.430689 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.430652 4285 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:34:21.433670 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.433647 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.433761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.433682 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.433761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.433711 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef8aa625af233e1fa89cb16cb4aca78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"ef8aa625af233e1fa89cb16cb4aca78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.436833 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.436815 4285 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.482330 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.482304 4285 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.482425 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.482376 4285 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.482920 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.482905 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeNotReady" Nov 02 21:34:21.482994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.482936 4285 setters.go:546] "Node became not ready" node="ip-10-0-131-47.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T21:34:21Z","lastTransitionTime":"2026-11-02T21:34:21Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 21:34:21.489981 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.489957 4285 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-47.us-west-1.compute.internal" is owned by different node (oldNodeID="50aa336b-bc8f-41b6-8d4a-f7e6fbf3b6d6", newNodeID=""), cleaning up... Nov 02 21:34:21.494257 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.494242 4285 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:21.494307 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.494280 4285 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.494307 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.494293 4285 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.533920 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.533827 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef8aa625af233e1fa89cb16cb4aca78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"ef8aa625af233e1fa89cb16cb4aca78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.533920 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.533873 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.533920 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.533892 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.534116 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.533939 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef8aa625af233e1fa89cb16cb4aca78e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"ef8aa625af233e1fa89cb16cb4aca78e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.534116 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.533946 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.534116 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.534004 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d7a9d6fa496da9c68538ee1b34b8e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" (UID: \"c9d7a9d6fa496da9c68538ee1b34b8e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.552449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.552425 4285 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:21.552560 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:21.552465 4285 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:21.748631 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.748604 4285 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-47.us-west-1.compute.internal" event="NodeReady" Nov 02 21:34:21.748766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:21.748678 4285 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:34:22.125305 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.125273 4285 apiserver.go:52] "Watching apiserver" Nov 02 21:34:22.196843 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.196808 4285 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:22.197000 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.196949 4285 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:22.227317 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.227293 4285 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:34:22.227655 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.227637 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf","openshift-dns/node-resolver-gkphg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal","openshift-multus/multus-8zxht","openshift-multus/multus-additional-cni-plugins-mv42m","openshift-cluster-node-tuning-operator/tuned-srw9t","openshift-image-registry/node-ca-9wp2q","openshift-multus/network-metrics-daemon-k75zj","openshift-network-diagnostics/network-check-target-gbwsv","openshift-network-operator/iptables-alerter-btpw2","kube-system/global-pull-secret-syncer-lmrxw","kube-system/konnectivity-agent-rtnjt","kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-8wg6d"] Nov 02 21:34:22.230013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.229991 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.236850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.236828 4285 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:34:22.238542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238525 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddrs\" (UniqueName: \"kubernetes.io/projected/295c327d-c476-4f98-84a1-a39dda118c73-kube-api-access-fddrs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.238588 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238550 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovn-node-metrics-cert\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.238588 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238569 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-modprobe-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.238588 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238583 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-kubernetes\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.238693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238601 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-systemd\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.238749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238706 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.238809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238791 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-systemd-units\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.238853 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238819 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-socket-dir-parent\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.238884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238864 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-kubelet\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.238919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238891 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-conf\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.238954 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238930 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-lib-modules\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.238954 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238944 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-hosts-file\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.239052 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238957 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-etc-kubernetes\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239052 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.238971 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da3003ef-ed3e-40ea-8dec-781b93a27cc2-host\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.239052 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239012 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sjh\" (UniqueName: \"kubernetes.io/projected/da3003ef-ed3e-40ea-8dec-781b93a27cc2-kube-api-access-b8sjh\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.239212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239049 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-cnibin\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.239212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239074 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.239212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239098 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-etc-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239134 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-k8s-cni-cncf-io\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239175 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-kubelet\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239205 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-hostroot\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239232 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-daemon-config\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239284 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-run\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.239375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239300 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2615d298-b340-465b-b9b0-f66aa0f59628-agent-certs\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:34:22.239375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239315 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-cnibin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239329 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.239375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239343 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-systemd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239552 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239383 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239552 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239415 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-system-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239552 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239437 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv79b\" (UniqueName: \"kubernetes.io/projected/3a1e59cc-2f43-4079-9713-b19528741a1c-kube-api-access-hv79b\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.239552 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239540 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:22.239725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239558 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.239725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239573 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-netns\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239588 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239603 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-config\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239617 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-device-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.239725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239666 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.239932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239752 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2615d298-b340-465b-b9b0-f66aa0f59628-konnectivity-ca\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:34:22.239932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239776 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-var-lib-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239794 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.239932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239815 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.239932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239849 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.239932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239905 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-ovn\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239937 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-log-socket\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239957 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-multus\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.240169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239979 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-multus-certs\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.240169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.239998 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8d2\" (UniqueName: \"kubernetes.io/projected/32d0f122-5c7a-45b3-868e-03157b31421a-kube-api-access-5t8d2\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.240169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240036 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgf2\" (UniqueName: \"kubernetes.io/projected/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-kube-api-access-mlgf2\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.240169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240114 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-kubelet-config\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.240337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240184 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-etc-tuned\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.240337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240231 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e761f1b3-b683-463e-9c05-6432569d4cd0-iptables-alerter-script\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.240337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240266 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.240337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240291 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-dbus\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.240337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240300 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-etc-tuned\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.240337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240316 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-bin\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240338 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-netd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240370 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-script-lib\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240415 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.240484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240445 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da3003ef-ed3e-40ea-8dec-781b93a27cc2-serviceca\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.240594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240485 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-env-overrides\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240513 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-conf-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.240594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240535 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-sys\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.240594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240561 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-var-lib-kubelet\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.240700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240597 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e761f1b3-b683-463e-9c05-6432569d4cd0-host-slash\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.240700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240615 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-slash\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240651 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-system-cni-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.240700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240677 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.240700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240695 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-node-log\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240827 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240721 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysconfig\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.240827 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240769 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.240827 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240795 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf29\" (UniqueName: \"kubernetes.io/projected/4321e24b-79fa-4817-a0eb-66f3d9c875be-kube-api-access-rzf29\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.240827 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240824 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-bin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.240928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240870 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4tv\" (UniqueName: \"kubernetes.io/projected/cfd48ab2-58a6-42c2-8077-3b9af1669069-kube-api-access-fq4tv\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.240928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240898 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-tmp-dir\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.240980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.240970 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.241030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241013 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-tmp-dir\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.241106 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241091 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-cni-binary-copy\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.241184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241115 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-netns\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.241184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241149 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.241184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241177 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-host\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.241316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241225 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-tmp\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.241316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241309 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d0f122-5c7a-45b3-868e-03157b31421a-tmp\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.241416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241367 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46x6\" (UniqueName: \"kubernetes.io/projected/e761f1b3-b683-463e-9c05-6432569d4cd0-kube-api-access-n46x6\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.241416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.241410 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvqd\" (UniqueName: \"kubernetes.io/projected/cd749222-a1c6-403b-bcfe-4897707b96cc-kube-api-access-zlvqd\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.273429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.273406 4285 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:22.273525 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:22.273459 4285 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:22.286645 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.286612 4285 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:34:22.286773 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:22.286673 4285 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-47.us-west-1.compute.internal" Nov 02 21:34:22.286997 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.286977 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95jdp\"" Nov 02 21:34:22.287088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287070 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:34:22.287186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287168 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.287235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287193 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f7c29\"" Nov 02 21:34:22.287367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287352 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:34:22.287434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287372 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.287544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287431 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:34:22.287544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287531 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:34:22.287544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287541 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287546 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287556 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287546 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287597 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287599 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287657 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287664 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsxz6\"" Nov 02 21:34:22.287681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287683 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287705 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287747 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2mch2\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287755 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287751 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287772 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287775 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287785 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287758 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287757 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nr5z4\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287830 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287831 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zd85x\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287852 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jhcsj\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287847 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sb4td\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287864 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287869 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b8lsz\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287885 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287887 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287912 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287910 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fbz7r\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.287970 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:34:22.288015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.288004 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qbt7b\"" Nov 02 21:34:22.289009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.288043 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:34:22.289009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.288096 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:34:22.289009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.288165 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:34:22.289009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.288216 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7clxp\"" Nov 02 21:34:22.289009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.289007 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovn-node-metrics-cert\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.289255 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.289187 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.289859 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.289837 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-config\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.289859 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.289854 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-daemon-config\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.289965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.289900 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/befb271e-da36-470a-ae6e-b7ce5dabf394-original-pull-secret\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.289965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.289946 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.290060 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.290046 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2615d298-b340-465b-b9b0-f66aa0f59628-konnectivity-ca\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:34:22.290818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.290802 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-env-overrides\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.290855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.290844 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e761f1b3-b683-463e-9c05-6432569d4cd0-iptables-alerter-script\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.290962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.290947 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a1e59cc-2f43-4079-9713-b19528741a1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.291018 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.291005 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4321e24b-79fa-4817-a0eb-66f3d9c875be-ovnkube-script-lib\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.291231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.291213 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da3003ef-ed3e-40ea-8dec-781b93a27cc2-serviceca\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.292109 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.292094 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd48ab2-58a6-42c2-8077-3b9af1669069-cni-binary-copy\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.298589 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.298574 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:34:22.299788 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.299773 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2615d298-b340-465b-b9b0-f66aa0f59628-agent-certs\") pod \"konnectivity-agent-rtnjt\" (UID: \"2615d298-b340-465b-b9b0-f66aa0f59628\") " pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:34:22.301825 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.301807 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgf2\" (UniqueName: \"kubernetes.io/projected/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-kube-api-access-mlgf2\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.301874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.301847 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv79b\" (UniqueName: \"kubernetes.io/projected/3a1e59cc-2f43-4079-9713-b19528741a1c-kube-api-access-hv79b\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.302060 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302044 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddrs\" (UniqueName: \"kubernetes.io/projected/295c327d-c476-4f98-84a1-a39dda118c73-kube-api-access-fddrs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.302674 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302646 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sjh\" (UniqueName: \"kubernetes.io/projected/da3003ef-ed3e-40ea-8dec-781b93a27cc2-kube-api-access-b8sjh\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.302816 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302794 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8d2\" (UniqueName: \"kubernetes.io/projected/32d0f122-5c7a-45b3-868e-03157b31421a-kube-api-access-5t8d2\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.302873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302831 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46x6\" (UniqueName: \"kubernetes.io/projected/e761f1b3-b683-463e-9c05-6432569d4cd0-kube-api-access-n46x6\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.302873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302798 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4tv\" (UniqueName: \"kubernetes.io/projected/cfd48ab2-58a6-42c2-8077-3b9af1669069-kube-api-access-fq4tv\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.302941 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302875 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvqd\" (UniqueName: \"kubernetes.io/projected/cd749222-a1c6-403b-bcfe-4897707b96cc-kube-api-access-zlvqd\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.302941 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302891 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf29\" (UniqueName: \"kubernetes.io/projected/4321e24b-79fa-4817-a0eb-66f3d9c875be-kube-api-access-rzf29\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.302941 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.302927 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j626w\" (UniqueName: \"kubernetes.io/projected/92bc027f-95b3-4c18-96b0-9d7bd6ab4413-kube-api-access-j626w\") pod \"network-check-target-gbwsv\" (UID: \"92bc027f-95b3-4c18-96b0-9d7bd6ab4413\") " pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:22.341692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341669 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-modprobe-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.341793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341696 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-kubernetes\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.341793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341713 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-systemd\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.341793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341728 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-systemd-units\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.341793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341745 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-socket-dir-parent\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.341793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341769 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-kubelet\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.341793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341789 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-conf\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341800 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-systemd-units\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341802 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-kubernetes\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341824 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-kubelet\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341824 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-socket-dir-parent\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341808 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-lib-modules\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341851 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-systemd\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341859 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-modprobe-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341872 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-hosts-file\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341883 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-lib-modules\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341889 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-etc-kubernetes\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341905 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da3003ef-ed3e-40ea-8dec-781b93a27cc2-host\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341917 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-etc-kubernetes\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341920 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-cnibin\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341923 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58637e1f-1378-4b0a-9c11-03f3fe45d6ae-hosts-file\") pod \"node-resolver-gkphg\" (UID: \"58637e1f-1378-4b0a-9c11-03f3fe45d6ae\") " pod="openshift-dns/node-resolver-gkphg" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341936 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341944 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da3003ef-ed3e-40ea-8dec-781b93a27cc2-host\") pod \"node-ca-9wp2q\" (UID: \"da3003ef-ed3e-40ea-8dec-781b93a27cc2\") " pod="openshift-image-registry/node-ca-9wp2q" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341949 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-conf\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341962 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-etc-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341965 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-cnibin\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.341994 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-etc-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342020 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-k8s-cni-cncf-io\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342065 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-kubelet\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342088 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-hostroot\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342106 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-k8s-cni-cncf-io\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342112 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-run\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342113 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-kubelet\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342159 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-hostroot\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342159 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-cnibin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342175 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-run\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342188 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342206 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-cnibin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342231 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-systemd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342252 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342278 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342286 4285 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342299 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342350 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-system-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342378 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-netns\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342405 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-system-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342421 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342447 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-device-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342308 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-systemd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342470 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-var-lib-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342516 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-netns\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342527 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342536 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-device-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342557 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342571 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342574 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-var-lib-openvswitch\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342573 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342604 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-cni-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342606 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342630 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-ovn\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342641 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342653 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-log-socket\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342673 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-multus\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342674 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-run-ovn\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342706 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-multus-certs\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342712 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-log-socket\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342733 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-kubelet-config\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342743 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-multus-certs\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342747 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-multus\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342755 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342785 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-dbus\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342794 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.342777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342802 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-bin\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342812 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-kubelet-config\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342818 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-netd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342837 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-netd\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342844 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-cni-bin\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342855 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342886 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-conf-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342829 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/befb271e-da36-470a-ae6e-b7ce5dabf394-dbus\") pod \"global-pull-secret-syncer-lmrxw\" (UID: \"befb271e-da36-470a-ae6e-b7ce5dabf394\") " pod="kube-system/global-pull-secret-syncer-lmrxw" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342890 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342908 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-sys\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342936 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-var-lib-kubelet\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342947 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-sys\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342912 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-multus-conf-dir\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342952 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e761f1b3-b683-463e-9c05-6432569d4cd0-host-slash\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343022 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-slash\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343049 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-system-cni-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342988 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-var-lib-kubelet\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343075 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-host-slash\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343075 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-node-log\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.342986 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e761f1b3-b683-463e-9c05-6432569d4cd0-host-slash\") pod \"iptables-alerter-btpw2\" (UID: \"e761f1b3-b683-463e-9c05-6432569d4cd0\") " pod="openshift-network-operator/iptables-alerter-btpw2" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343093 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a1e59cc-2f43-4079-9713-b19528741a1c-system-cni-dir\") pod \"multus-additional-cni-plugins-mv42m\" (UID: \"3a1e59cc-2f43-4079-9713-b19528741a1c\") " pod="openshift-multus/multus-additional-cni-plugins-mv42m" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343115 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4321e24b-79fa-4817-a0eb-66f3d9c875be-node-log\") pod \"ovnkube-node-8wg6d\" (UID: \"4321e24b-79fa-4817-a0eb-66f3d9c875be\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343140 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysconfig\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343172 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysconfig\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343174 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343205 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-bin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343220 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343225 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343244 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-var-lib-cni-bin\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343260 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-netns\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343287 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343294 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd749222-a1c6-403b-bcfe-4897707b96cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k5gtf\" (UID: \"cd749222-a1c6-403b-bcfe-4897707b96cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k5gtf" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343308 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-host\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343340 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-host\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343318 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfd48ab2-58a6-42c2-8077-3b9af1669069-host-run-netns\") pod \"multus-8zxht\" (UID: \"cfd48ab2-58a6-42c2-8077-3b9af1669069\") " pod="openshift-multus/multus-8zxht" Nov 02 21:34:22.344191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.343387 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32d0f122-5c7a-45b3-868e-03157b31421a-etc-sysctl-d\") pod \"tuned-srw9t\" (UID: \"32d0f122-5c7a-45b3-868e-03157b31421a\") " pod="openshift-cluster-node-tuning-operator/tuned-srw9t" Nov 02 21:34:22.345069 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.344478 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/295c327d-c476-4f98-84a1-a39dda118c73-metrics-certs\") pod \"network-metrics-daemon-k75zj\" (UID: \"295c327d-c476-4f98-84a1-a39dda118c73\") " pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.489941 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.489858 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:22.530356 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.530328 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k75zj" Nov 02 21:34:22.612331 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.612309 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:22.689100 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.689063 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:22.880277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.880204 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:22.934394 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.934351 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k75zj"] Nov 02 21:34:22.938388 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:22.938350 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295c327d_c476_4f98_84a1_a39dda118c73.slice/crio-5ee286de5181706a841035c78914cfaf0fb2f16a230c2581613dd68a43a667a3 WatchSource:0}: Error finding container 5ee286de5181706a841035c78914cfaf0fb2f16a230c2581613dd68a43a667a3: Status 404 returned error can't find the container with id 5ee286de5181706a841035c78914cfaf0fb2f16a230c2581613dd68a43a667a3 Nov 02 21:34:22.940515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:22.940498 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:34:23.035612 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:23.035580 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:23.186148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:23.186064 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:23.199608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:23.199572 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k75zj" event={"ID":"295c327d-c476-4f98-84a1-a39dda118c73","Type":"ContainerStarted","Data":"5ee286de5181706a841035c78914cfaf0fb2f16a230c2581613dd68a43a667a3"} Nov 02 21:34:23.481139 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:23.481053 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:25.203103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:25.203075 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:25.205641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:25.205613 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k75zj" event={"ID":"295c327d-c476-4f98-84a1-a39dda118c73","Type":"ContainerStarted","Data":"b3a2cee9ad5fc942190fc33aec65f215599ff2a9555290cbd9ef4194fa2a7af2"} Nov 02 21:34:25.205757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:25.205669 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k75zj" event={"ID":"295c327d-c476-4f98-84a1-a39dda118c73","Type":"ContainerStarted","Data":"6a3cbd3989191c4ad2fc9667a864aa5d0426cc0158d4fab319b5b7fe2a7ddeb1"} Nov 02 21:34:25.351073 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:25.351040 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:26.965730 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:26.965690 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:27.797029 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:27.796998 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:27.797294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:27.797277 4285 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:34:30.609434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.609403 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8"] Nov 02 21:34:30.612113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.612090 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v9qdt"] Nov 02 21:34:30.612253 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.612209 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:30.613741 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.613723 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-276px"] Nov 02 21:34:30.613873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.613854 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.615535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.615518 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.667923 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.667889 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zntk7\"" Nov 02 21:34:30.667923 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.667919 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 21:34:30.670982 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.670961 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:34:30.671658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.671641 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8"] Nov 02 21:34:30.672164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.672152 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:34:30.693418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693392 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e4ccef0-1600-4f94-883b-af10ec8699e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hwpp8\" (UID: \"9e4ccef0-1600-4f94-883b-af10ec8699e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:30.693505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693425 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r92k\" (UniqueName: \"kubernetes.io/projected/90545560-29ab-4188-9e10-52ade1441347-kube-api-access-9r92k\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.693505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693445 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90545560-29ab-4188-9e10-52ade1441347-crio-socket\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.693505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693461 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04ccdb32-f19e-44cc-870b-3f43b3f4d8f1-cert\") pod \"ingress-canary-v9qdt\" (UID: \"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1\") " pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.693601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693567 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90545560-29ab-4188-9e10-52ade1441347-data-volume\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.693601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693592 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90545560-29ab-4188-9e10-52ade1441347-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.693661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693610 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90545560-29ab-4188-9e10-52ade1441347-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.693702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.693687 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wtc\" (UniqueName: \"kubernetes.io/projected/04ccdb32-f19e-44cc-870b-3f43b3f4d8f1-kube-api-access-p8wtc\") pod \"ingress-canary-v9qdt\" (UID: \"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1\") " pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.717564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.717540 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:34:30.717883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.717800 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:34:30.718063 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.718043 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:34:30.718163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.718066 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:34:30.719557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.719532 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5898cd6f4-cg4gf"] Nov 02 21:34:30.721906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.721888 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m4tmg"] Nov 02 21:34:30.722030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.722013 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.724320 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.724302 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.724792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.724487 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-276px"] Nov 02 21:34:30.768115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.768091 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-stcz5\"" Nov 02 21:34:30.768275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.768260 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rwnh\"" Nov 02 21:34:30.768353 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.768341 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:34:30.772934 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.772903 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v9qdt"] Nov 02 21:34:30.794617 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794592 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1f838e1-8290-4c10-951c-e2725201c0c2-metrics-tls\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.794759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794634 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e4ccef0-1600-4f94-883b-af10ec8699e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hwpp8\" (UID: \"9e4ccef0-1600-4f94-883b-af10ec8699e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:30.794759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794666 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-image-registry-private-configuration\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.794852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794765 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r92k\" (UniqueName: \"kubernetes.io/projected/90545560-29ab-4188-9e10-52ade1441347-kube-api-access-9r92k\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.794852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794802 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90545560-29ab-4188-9e10-52ade1441347-crio-socket\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.794852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794823 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04ccdb32-f19e-44cc-870b-3f43b3f4d8f1-cert\") pod \"ingress-canary-v9qdt\" (UID: \"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1\") " pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.794993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794848 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-installation-pull-secrets\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.794993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794891 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90545560-29ab-4188-9e10-52ade1441347-data-volume\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.794993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794901 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90545560-29ab-4188-9e10-52ade1441347-crio-socket\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.794993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794944 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-ca-trust-extracted\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.794993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.794972 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-registry-certificates\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795023 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtk9\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-kube-api-access-hxtk9\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795068 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90545560-29ab-4188-9e10-52ade1441347-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795091 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-bound-sa-token\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795109 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6m4s\" (UniqueName: \"kubernetes.io/projected/b1f838e1-8290-4c10-951c-e2725201c0c2-kube-api-access-f6m4s\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795229 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90545560-29ab-4188-9e10-52ade1441347-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795251 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90545560-29ab-4188-9e10-52ade1441347-data-volume\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.795274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795265 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-registry-tls\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.795633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795309 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wtc\" (UniqueName: \"kubernetes.io/projected/04ccdb32-f19e-44cc-870b-3f43b3f4d8f1-kube-api-access-p8wtc\") pod \"ingress-canary-v9qdt\" (UID: \"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1\") " pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.795633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795343 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-trusted-ca\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.795633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795368 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f838e1-8290-4c10-951c-e2725201c0c2-config-volume\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.795633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795499 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f838e1-8290-4c10-951c-e2725201c0c2-tmp-dir\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.795778 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.795757 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90545560-29ab-4188-9e10-52ade1441347-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.797309 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.797288 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90545560-29ab-4188-9e10-52ade1441347-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.797408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.797313 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e4ccef0-1600-4f94-883b-af10ec8699e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hwpp8\" (UID: \"9e4ccef0-1600-4f94-883b-af10ec8699e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:30.797449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.797410 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04ccdb32-f19e-44cc-870b-3f43b3f4d8f1-cert\") pod \"ingress-canary-v9qdt\" (UID: \"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1\") " pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.828442 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.828416 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:34:30.828774 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.828756 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 21:34:30.829174 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.829155 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4tmg"] Nov 02 21:34:30.829577 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.829559 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5898cd6f4-cg4gf"] Nov 02 21:34:30.851730 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.851703 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:30.851837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.851769 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wtc\" (UniqueName: \"kubernetes.io/projected/04ccdb32-f19e-44cc-870b-3f43b3f4d8f1-kube-api-access-p8wtc\") pod \"ingress-canary-v9qdt\" (UID: \"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1\") " pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.851899 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.851871 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r92k\" (UniqueName: \"kubernetes.io/projected/90545560-29ab-4188-9e10-52ade1441347-kube-api-access-9r92k\") pod \"insights-runtime-extractor-276px\" (UID: \"90545560-29ab-4188-9e10-52ade1441347\") " pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.868164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.868102 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:34:30.868164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.868110 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plzkt\"" Nov 02 21:34:30.868164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.868102 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 21:34:30.868312 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.868172 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 21:34:30.868312 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.868234 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 21:34:30.868312 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.868277 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jfs7n\"" Nov 02 21:34:30.895792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895766 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1f838e1-8290-4c10-951c-e2725201c0c2-metrics-tls\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.895883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895801 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-image-registry-private-configuration\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.895883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895833 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-installation-pull-secrets\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.895883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895866 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-ca-trust-extracted\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895882 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-registry-certificates\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895900 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtk9\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-kube-api-access-hxtk9\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895923 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-bound-sa-token\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895945 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6m4s\" (UniqueName: \"kubernetes.io/projected/b1f838e1-8290-4c10-951c-e2725201c0c2-kube-api-access-f6m4s\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.896023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.895972 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-registry-tls\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896007 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-trusted-ca\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896172 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f838e1-8290-4c10-951c-e2725201c0c2-config-volume\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.896328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896212 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f838e1-8290-4c10-951c-e2725201c0c2-tmp-dir\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.896328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896308 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-ca-trust-extracted\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.896632 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896573 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f838e1-8290-4c10-951c-e2725201c0c2-tmp-dir\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.896886 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896864 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f838e1-8290-4c10-951c-e2725201c0c2-config-volume\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.897015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.896943 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-registry-certificates\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.897391 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.897367 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-trusted-ca\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.898490 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.898466 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-installation-pull-secrets\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.898592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.898466 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-image-registry-private-configuration\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.898592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.898516 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-registry-tls\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.898758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.898738 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1f838e1-8290-4c10-951c-e2725201c0c2-metrics-tls\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:30.922622 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.922595 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:30.929219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.929205 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v9qdt" Nov 02 21:34:30.935378 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.935356 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-276px" Nov 02 21:34:30.952638 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.952587 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-bound-sa-token\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.953369 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.953172 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtk9\" (UniqueName: \"kubernetes.io/projected/90112989-dfa7-4d74-a0cf-e7da4a3f1bf5-kube-api-access-hxtk9\") pod \"image-registry-5898cd6f4-cg4gf\" (UID: \"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5\") " pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:30.953369 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:30.953316 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6m4s\" (UniqueName: \"kubernetes.io/projected/b1f838e1-8290-4c10-951c-e2725201c0c2-kube-api-access-f6m4s\") pod \"dns-default-m4tmg\" (UID: \"b1f838e1-8290-4c10-951c-e2725201c0c2\") " pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:31.003001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.002816 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:31.034511 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.034488 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:31.040064 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.040042 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:31.289659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.289628 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:31.294108 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.294082 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v9qdt"] Nov 02 21:34:31.297728 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:31.297702 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ccdb32_f19e_44cc_870b_3f43b3f4d8f1.slice/crio-285c5c8fbe22859b4d6e3fa6106349546a68df95aa7c3b79c56bf1e6d46ecada WatchSource:0}: Error finding container 285c5c8fbe22859b4d6e3fa6106349546a68df95aa7c3b79c56bf1e6d46ecada: Status 404 returned error can't find the container with id 285c5c8fbe22859b4d6e3fa6106349546a68df95aa7c3b79c56bf1e6d46ecada Nov 02 21:34:31.333600 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.333574 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8"] Nov 02 21:34:31.336500 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:31.336472 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4ccef0_1600_4f94_883b_af10ec8699e7.slice/crio-8163b8c130bc9724ce1732697ed37fd586b7fcf91e67c67ad5144aaefb1bac33 WatchSource:0}: Error finding container 8163b8c130bc9724ce1732697ed37fd586b7fcf91e67c67ad5144aaefb1bac33: Status 404 returned error can't find the container with id 8163b8c130bc9724ce1732697ed37fd586b7fcf91e67c67ad5144aaefb1bac33 Nov 02 21:34:31.344539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.344494 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-276px"] Nov 02 21:34:31.346677 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:31.346654 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90545560_29ab_4188_9e10_52ade1441347.slice/crio-d66875cc7d6ac632b9ffaa2470c0c086e05a8bbc7c71186f2c877df8279f3c6c WatchSource:0}: Error finding container d66875cc7d6ac632b9ffaa2470c0c086e05a8bbc7c71186f2c877df8279f3c6c: Status 404 returned error can't find the container with id d66875cc7d6ac632b9ffaa2470c0c086e05a8bbc7c71186f2c877df8279f3c6c Nov 02 21:34:31.401057 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.401035 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5898cd6f4-cg4gf"] Nov 02 21:34:31.404052 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:31.404027 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90112989_dfa7_4d74_a0cf_e7da4a3f1bf5.slice/crio-f1f102919b4cf05dde31905cede73ac1bf5d289724bf74f50868469fd846bd5e WatchSource:0}: Error finding container f1f102919b4cf05dde31905cede73ac1bf5d289724bf74f50868469fd846bd5e: Status 404 returned error can't find the container with id f1f102919b4cf05dde31905cede73ac1bf5d289724bf74f50868469fd846bd5e Nov 02 21:34:31.412168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.412048 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4tmg"] Nov 02 21:34:31.415993 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:31.415942 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f838e1_8290_4c10_951c_e2725201c0c2.slice/crio-1b3e91e160f3ce7006ca03cbb4e3d9fd5f577dee64ea8be5b886506ecdd13138 WatchSource:0}: Error finding container 1b3e91e160f3ce7006ca03cbb4e3d9fd5f577dee64ea8be5b886506ecdd13138: Status 404 returned error can't find the container with id 1b3e91e160f3ce7006ca03cbb4e3d9fd5f577dee64ea8be5b886506ecdd13138 Nov 02 21:34:31.438585 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:31.438555 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:32.222632 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.222593 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4tmg" event={"ID":"b1f838e1-8290-4c10-951c-e2725201c0c2","Type":"ContainerStarted","Data":"1b3e91e160f3ce7006ca03cbb4e3d9fd5f577dee64ea8be5b886506ecdd13138"} Nov 02 21:34:32.224575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.224544 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" event={"ID":"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5","Type":"ContainerStarted","Data":"5a23f932d900cbfd6af3971b961c8fe5abc335284235f7ad67603a646e96a132"} Nov 02 21:34:32.224753 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.224736 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" event={"ID":"90112989-dfa7-4d74-a0cf-e7da4a3f1bf5","Type":"ContainerStarted","Data":"f1f102919b4cf05dde31905cede73ac1bf5d289724bf74f50868469fd846bd5e"} Nov 02 21:34:32.226417 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.226390 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-276px" event={"ID":"90545560-29ab-4188-9e10-52ade1441347","Type":"ContainerStarted","Data":"5dc322113862a2f8e3304788df7e09f0666ecd61dd68d2d46a283cb944c729a4"} Nov 02 21:34:32.226542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.226446 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-276px" event={"ID":"90545560-29ab-4188-9e10-52ade1441347","Type":"ContainerStarted","Data":"d66875cc7d6ac632b9ffaa2470c0c086e05a8bbc7c71186f2c877df8279f3c6c"} Nov 02 21:34:32.227828 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.227801 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v9qdt" event={"ID":"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1","Type":"ContainerStarted","Data":"285c5c8fbe22859b4d6e3fa6106349546a68df95aa7c3b79c56bf1e6d46ecada"} Nov 02 21:34:32.229100 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.229080 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" event={"ID":"9e4ccef0-1600-4f94-883b-af10ec8699e7","Type":"ContainerStarted","Data":"8163b8c130bc9724ce1732697ed37fd586b7fcf91e67c67ad5144aaefb1bac33"} Nov 02 21:34:32.340824 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.340763 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" podStartSLOduration=2.340742315 podStartE2EDuration="2.340742315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:32.33948402 +0000 UTC m=+11.301034355" watchObservedRunningTime="2026-11-02 21:34:32.340742315 +0000 UTC m=+11.302292633" Nov 02 21:34:32.712267 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:32.712231 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:33.244661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:33.244621 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-276px" event={"ID":"90545560-29ab-4188-9e10-52ade1441347","Type":"ContainerStarted","Data":"68558dd7e91d66af2c5521d13b0bd1e4440fb765c6c9fe09073be41bf2ddb018"} Nov 02 21:34:33.250759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:33.246574 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" event={"ID":"9e4ccef0-1600-4f94-883b-af10ec8699e7","Type":"ContainerStarted","Data":"c7104a9b455b8af6d2db9f73e26f9cae03dee05637c2f32a685510bbf8f1c1e3"} Nov 02 21:34:33.250759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:33.246873 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:33.252175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:33.252111 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" Nov 02 21:34:33.357906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:33.357853 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hwpp8" podStartSLOduration=1.774374999 podStartE2EDuration="3.357839893s" totalImagesPullingTime="1.583464894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:33.357477579 +0000 UTC m=+12.319027902" watchObservedRunningTime="2026-11-02 21:34:33.357839893 +0000 UTC m=+12.319390212" Nov 02 21:34:34.253663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.253624 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4tmg" event={"ID":"b1f838e1-8290-4c10-951c-e2725201c0c2","Type":"ContainerStarted","Data":"b24808957cff5d85aa58a6bd32f1ae699aefd0a2f93ac0125bd279fadec96494"} Nov 02 21:34:34.254063 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.253683 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4tmg" event={"ID":"b1f838e1-8290-4c10-951c-e2725201c0c2","Type":"ContainerStarted","Data":"1f8d279e9dbd7ff6f7eded0a54ec1c4850a69ad56d5af6e51386391f31be7c85"} Nov 02 21:34:34.254137 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.254068 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:34.355641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.355606 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:34.512602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.512526 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:34.658882 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.658858 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:34.736775 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.736403 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-m4tmg" podStartSLOduration=2.334796537 podStartE2EDuration="4.736382489s" totalImagesPullingTime="2.401585952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:34.367521851 +0000 UTC m=+13.329072170" watchObservedRunningTime="2026-11-02 21:34:34.736382489 +0000 UTC m=+13.697932812" Nov 02 21:34:34.736775 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.736565 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh"] Nov 02 21:34:34.739842 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.739820 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.793230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.793143 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 21:34:34.793230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.793203 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:34:34.811922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.811890 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:34.839052 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.839015 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvml\" (UniqueName: \"kubernetes.io/projected/54ee2947-1446-4a7f-a3e6-af59743823e3-kube-api-access-6cvml\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.839238 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.839064 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54ee2947-1446-4a7f-a3e6-af59743823e3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.839238 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.839211 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ee2947-1446-4a7f-a3e6-af59743823e3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.839333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.839250 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ee2947-1446-4a7f-a3e6-af59743823e3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.842758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.842733 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 21:34:34.842840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.842763 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:34:34.842893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.842806 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zqxqp\"" Nov 02 21:34:34.842893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.842849 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:34:34.843532 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.843510 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh"] Nov 02 21:34:34.940044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.939992 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ee2947-1446-4a7f-a3e6-af59743823e3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.940044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.940052 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ee2947-1446-4a7f-a3e6-af59743823e3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.940335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.940081 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvml\" (UniqueName: \"kubernetes.io/projected/54ee2947-1446-4a7f-a3e6-af59743823e3-kube-api-access-6cvml\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.940335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.940109 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54ee2947-1446-4a7f-a3e6-af59743823e3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.940772 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.940750 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ee2947-1446-4a7f-a3e6-af59743823e3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.942551 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.942529 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ee2947-1446-4a7f-a3e6-af59743823e3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.942698 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.942678 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54ee2947-1446-4a7f-a3e6-af59743823e3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:34.973242 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.973214 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rtnjt" Nov 02 21:34:34.996666 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:34.996635 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvml\" (UniqueName: \"kubernetes.io/projected/54ee2947-1446-4a7f-a3e6-af59743823e3-kube-api-access-6cvml\") pod \"prometheus-operator-9cd89b74c-vv9jh\" (UID: \"54ee2947-1446-4a7f-a3e6-af59743823e3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:35.050465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:35.050380 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" Nov 02 21:34:35.260244 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:35.260210 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-276px" event={"ID":"90545560-29ab-4188-9e10-52ade1441347","Type":"ContainerStarted","Data":"2e33eee956499dbf62071c50a6c6aaa2e276359ac30955cc85ab495faf139b6f"} Nov 02 21:34:35.261710 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:35.261684 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v9qdt" event={"ID":"04ccdb32-f19e-44cc-870b-3f43b3f4d8f1","Type":"ContainerStarted","Data":"c41dc1472c744708ac7aa87f53ad936660fba4ed177ecc6ed245310678d82207"} Nov 02 21:34:35.375316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:35.375215 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-276px" podStartSLOduration=2.262662322 podStartE2EDuration="5.37519993s" totalImagesPullingTime="3.112537608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:34:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:35.374342786 +0000 UTC m=+14.335893105" watchObservedRunningTime="2026-11-02 21:34:35.37519993 +0000 UTC m=+14.336750249" Nov 02 21:34:35.412730 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:35.412696 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh"] Nov 02 21:34:35.447977 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:35.447949 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ee2947_1446_4a7f_a3e6_af59743823e3.slice/crio-91c872863940e2ff292e07779329f2a709a76a47ff17535e10638859720aae65 WatchSource:0}: Error finding container 91c872863940e2ff292e07779329f2a709a76a47ff17535e10638859720aae65: Status 404 returned error can't find the container with id 91c872863940e2ff292e07779329f2a709a76a47ff17535e10638859720aae65 Nov 02 21:34:35.486545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:35.486496 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v9qdt" podStartSLOduration=2.136838174 podStartE2EDuration="5.486482475s" totalImagesPullingTime="3.349644301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:35.485911081 +0000 UTC m=+14.447461400" watchObservedRunningTime="2026-11-02 21:34:35.486482475 +0000 UTC m=+14.448032792" Nov 02 21:34:36.145518 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:36.145485 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:36.265755 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:36.265712 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" event={"ID":"54ee2947-1446-4a7f-a3e6-af59743823e3","Type":"ContainerStarted","Data":"91c872863940e2ff292e07779329f2a709a76a47ff17535e10638859720aae65"} Nov 02 21:34:37.271495 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:37.271464 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" event={"ID":"54ee2947-1446-4a7f-a3e6-af59743823e3","Type":"ContainerStarted","Data":"050085c77ca61b90bbe95a170be6ff10339975536480dca3be49226257ab42e1"} Nov 02 21:34:37.271863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:37.271516 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" event={"ID":"54ee2947-1446-4a7f-a3e6-af59743823e3","Type":"ContainerStarted","Data":"c508d25155a496917822469bbedc4fe06bd1e19415ac694ae30fc245f00094b3"} Nov 02 21:34:37.420736 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:37.420616 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vv9jh" podStartSLOduration=1.719086648 podStartE2EDuration="3.420585375s" totalImagesPullingTime="1.701498727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:37.419827861 +0000 UTC m=+16.381378181" watchObservedRunningTime="2026-11-02 21:34:37.420585375 +0000 UTC m=+16.382135694" Nov 02 21:34:39.655535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:39.655502 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:39.879350 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:39.879310 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:40.030280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.030244 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:40.216264 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.216218 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:40.369470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.369390 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:40.467296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.467264 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xrc7x"] Nov 02 21:34:40.469504 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.469490 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.537486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.537460 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:40.573197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.573160 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:34:40.573197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.573194 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xsz5x\"" Nov 02 21:34:40.573407 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.573160 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:34:40.573407 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.573235 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:34:40.579382 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579359 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-sys\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579518 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579392 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx85\" (UniqueName: \"kubernetes.io/projected/2e2fee67-70fc-4aec-9357-cce7e8461ef3-kube-api-access-sdx85\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579518 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579412 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-tls\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579518 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579439 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-wtmp\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579518 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579484 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579701 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579559 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fee67-70fc-4aec-9357-cce7e8461ef3-metrics-client-ca\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579701 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579580 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-root\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579701 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579604 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-textfile\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.579701 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.579619 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.680561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680470 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-wtmp\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.680561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680536 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680577 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fee67-70fc-4aec-9357-cce7e8461ef3-metrics-client-ca\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680608 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-root\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680623 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-wtmp\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680635 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-textfile\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680664 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680684 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-root\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680733 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-sys\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680766 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx85\" (UniqueName: \"kubernetes.io/projected/2e2fee67-70fc-4aec-9357-cce7e8461ef3-kube-api-access-sdx85\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680793 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-tls\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.680804 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e2fee67-70fc-4aec-9357-cce7e8461ef3-sys\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681468 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.681054 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-textfile\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681468 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.681176 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.681468 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.681239 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e2fee67-70fc-4aec-9357-cce7e8461ef3-metrics-client-ca\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.682956 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.682935 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.683150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.683112 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e2fee67-70fc-4aec-9357-cce7e8461ef3-node-exporter-tls\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.737001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.736974 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx85\" (UniqueName: \"kubernetes.io/projected/2e2fee67-70fc-4aec-9357-cce7e8461ef3-kube-api-access-sdx85\") pod \"node-exporter-xrc7x\" (UID: \"2e2fee67-70fc-4aec-9357-cce7e8461ef3\") " pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.780678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.780646 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xrc7x" Nov 02 21:34:40.789878 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:40.789850 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2fee67_70fc_4aec_9357_cce7e8461ef3.slice/crio-e383073bb22a78477df741a62ab04b348e1d585eb67aa73b8635373658fe674a WatchSource:0}: Error finding container e383073bb22a78477df741a62ab04b348e1d585eb67aa73b8635373658fe674a: Status 404 returned error can't find the container with id e383073bb22a78477df741a62ab04b348e1d585eb67aa73b8635373658fe674a Nov 02 21:34:40.809383 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:40.809351 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:41.031079 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:41.031044 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:41.285074 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:41.284994 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrc7x" event={"ID":"2e2fee67-70fc-4aec-9357-cce7e8461ef3","Type":"ContainerStarted","Data":"e383073bb22a78477df741a62ab04b348e1d585eb67aa73b8635373658fe674a"} Nov 02 21:34:41.760115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:41.760081 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:41.912903 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:41.912874 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:42.060763 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.060688 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:42.208740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.208713 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:42.288923 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.288895 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrc7x" event={"ID":"2e2fee67-70fc-4aec-9357-cce7e8461ef3","Type":"ContainerStarted","Data":"dc81786620eae93a5eac0d306bf53abf16494fc7f894accc038d2b6527de0dc1"} Nov 02 21:34:42.601944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.601908 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:42.613850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.613781 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:34:42.616031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.616014 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.668610 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.668584 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:34:42.694984 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.694959 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.694989 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695010 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695031 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695156 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695250 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695277 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz2z\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695412 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695302 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695412 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695359 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695433 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695464 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695609 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695520 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.695609 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.695551 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.718144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718102 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:34:42.718144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718108 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:34:42.718144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718141 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:34:42.718419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718145 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:34:42.718419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718164 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:34:42.718419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718402 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:34:42.718626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718406 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:34:42.718626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.718440 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xjmjs\"" Nov 02 21:34:42.767885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.767847 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:34:42.768927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.768909 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:34:42.796086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796057 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796086 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796112 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796154 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796176 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796202 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796228 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796268 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796293 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796333 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796349 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz2z\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796369 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796394 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.796803 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.796566 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.798396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.797359 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.798396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.798346 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.799406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.799359 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.799527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.799494 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.799713 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.799676 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.800294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.799817 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.800294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.799940 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.800294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.800251 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.800527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.800506 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.800582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.800568 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.801001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.800984 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.853840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.853806 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz2z\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z\") pod \"alertmanager-main-0\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:42.930782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:42.930707 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:34:43.293863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:43.293832 4285 generic.go:356] "Generic (PLEG): container finished" podID="2e2fee67-70fc-4aec-9357-cce7e8461ef3" containerID="dc81786620eae93a5eac0d306bf53abf16494fc7f894accc038d2b6527de0dc1" exitCode=0 Nov 02 21:34:43.294045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:43.293869 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrc7x" event={"ID":"2e2fee67-70fc-4aec-9357-cce7e8461ef3","Type":"ContainerDied","Data":"dc81786620eae93a5eac0d306bf53abf16494fc7f894accc038d2b6527de0dc1"} Nov 02 21:34:43.401468 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:43.399376 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:34:43.404205 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:43.404180 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06b40f1_34b1_47a8_8194_ebda81966a22.slice/crio-24a56f3c3fdc302f9caadd9c30f4af8397be28444dbc5368c53138b3c0d82014 WatchSource:0}: Error finding container 24a56f3c3fdc302f9caadd9c30f4af8397be28444dbc5368c53138b3c0d82014: Status 404 returned error can't find the container with id 24a56f3c3fdc302f9caadd9c30f4af8397be28444dbc5368c53138b3c0d82014 Nov 02 21:34:44.299337 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.299297 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"24a56f3c3fdc302f9caadd9c30f4af8397be28444dbc5368c53138b3c0d82014"} Nov 02 21:34:44.302517 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.302484 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrc7x" event={"ID":"2e2fee67-70fc-4aec-9357-cce7e8461ef3","Type":"ContainerStarted","Data":"5f42f29b309342696301899860900a96dd1b7c2bdc48201823d0e1c4dcb93c66"} Nov 02 21:34:44.302666 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.302542 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xrc7x" event={"ID":"2e2fee67-70fc-4aec-9357-cce7e8461ef3","Type":"ContainerStarted","Data":"18197c6d401b79e91f16ea157053aaa2498ce4bf6d8681c5b7f23492869ff718"} Nov 02 21:34:44.383532 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.383502 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:44.436361 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.436308 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xrc7x" podStartSLOduration=2.929551421 podStartE2EDuration="4.436291758s" totalImagesPullingTime="1.440067142s" totalInitContainerRuntime="66.673195ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:44.43597497 +0000 UTC m=+23.397525287" watchObservedRunningTime="2026-11-02 21:34:44.436291758 +0000 UTC m=+23.397842114" Nov 02 21:34:44.542664 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.542633 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:44.738611 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.738581 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:44.941781 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:44.941733 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:45.096091 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.096002 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:45.248429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.248395 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:45.267412 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.267385 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m4tmg" Nov 02 21:34:45.307097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.307062 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c" exitCode=0 Nov 02 21:34:45.307468 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.307148 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c"} Nov 02 21:34:45.331339 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.330889 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dc98b96fb-mslq5"] Nov 02 21:34:45.332718 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.332703 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.384440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.384373 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 21:34:45.384440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.384402 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xxx78\"" Nov 02 21:34:45.395102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.395078 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dc98b96fb-mslq5"] Nov 02 21:34:45.409591 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.409560 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 21:34:45.409720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.409596 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 21:34:45.419905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.419876 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-client-ca-bundle\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.420035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.419927 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-secret-metrics-server-tls\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.420035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.420012 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-metrics-server-audit-profiles\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.420156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.420037 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-secret-metrics-server-client-certs\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.420281 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.420247 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.420507 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.420477 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-audit-log\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.420759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.420738 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmvc\" (UniqueName: \"kubernetes.io/projected/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-kube-api-access-tgmvc\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.484655 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.484624 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 21:34:45.484655 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.484639 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1850m2vhs0i7b\"" Nov 02 21:34:45.521482 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521452 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-audit-log\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521491 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmvc\" (UniqueName: \"kubernetes.io/projected/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-kube-api-access-tgmvc\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521521 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-client-ca-bundle\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521555 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-secret-metrics-server-tls\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521598 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-metrics-server-audit-profiles\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521620 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-secret-metrics-server-client-certs\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521833 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.521919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.521898 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-audit-log\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.522712 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.522684 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-metrics-server-audit-profiles\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.522796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.522680 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.524186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.524167 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-secret-metrics-server-client-certs\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.524266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.524180 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-secret-metrics-server-tls\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.524266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.524193 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-client-ca-bundle\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.577103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.577078 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmvc\" (UniqueName: \"kubernetes.io/projected/8b619304-95b7-43ce-b2f2-5b7d6e37ed59-kube-api-access-tgmvc\") pod \"metrics-server-6dc98b96fb-mslq5\" (UID: \"8b619304-95b7-43ce-b2f2-5b7d6e37ed59\") " pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.642800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.642730 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:34:45.680626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.680597 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:45.836302 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:45.836265 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:46.004652 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:34:46.004611 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:34:46.055655 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:46.055622 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dc98b96fb-mslq5"] Nov 02 21:34:46.058480 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:46.058449 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b619304_95b7_43ce_b2f2_5b7d6e37ed59.slice/crio-c708e948fe4675a03d689e4c4260941fa251245d5cef05df443edba562021d26 WatchSource:0}: Error finding container c708e948fe4675a03d689e4c4260941fa251245d5cef05df443edba562021d26: Status 404 returned error can't find the container with id c708e948fe4675a03d689e4c4260941fa251245d5cef05df443edba562021d26 Nov 02 21:34:46.303321 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:46.303231 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wg6d" Nov 02 21:34:46.317144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:46.317088 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" event={"ID":"8b619304-95b7-43ce-b2f2-5b7d6e37ed59","Type":"ContainerStarted","Data":"c708e948fe4675a03d689e4c4260941fa251245d5cef05df443edba562021d26"} Nov 02 21:34:47.323571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:47.323542 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640"} Nov 02 21:34:47.323926 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:47.323607 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a"} Nov 02 21:34:47.323926 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:47.323654 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060"} Nov 02 21:34:47.323926 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:47.323677 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9"} Nov 02 21:34:48.332015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:48.331926 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6"} Nov 02 21:34:48.334282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:48.334247 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" event={"ID":"8b619304-95b7-43ce-b2f2-5b7d6e37ed59","Type":"ContainerStarted","Data":"490d8b3f54b609928fe3b96c0094cd8ebfbcaef53b50add6575df61689752da9"} Nov 02 21:34:48.448794 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:48.448727 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" podStartSLOduration=1.427912383 podStartE2EDuration="3.448707571s" totalImagesPullingTime="2.020795188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:48.447423728 +0000 UTC m=+27.408974051" watchObservedRunningTime="2026-11-02 21:34:48.448707571 +0000 UTC m=+27.410257891" Nov 02 21:34:49.249985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:49.249949 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:49.341602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:49.341565 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerStarted","Data":"5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d"} Nov 02 21:34:49.612022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:49.611918 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.373842888 podStartE2EDuration="7.611904299s" totalImagesPullingTime="4.226775071s" totalInitContainerRuntime="11.28634ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:49.56202469 +0000 UTC m=+28.523575038" watchObservedRunningTime="2026-11-02 21:34:49.611904299 +0000 UTC m=+28.573454711" Nov 02 21:34:54.258546 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:54.258513 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5898cd6f4-cg4gf" Nov 02 21:34:54.562406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:54.562333 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:54.719680 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:54.719644 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:54.900785 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:54.900676 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:55.102527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:55.102444 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:55.516866 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:55.516837 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:55.684780 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:55.684749 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:55.857672 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:55.857593 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:56.253442 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.253402 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5vwbc"] Nov 02 21:34:56.255954 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.255933 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.311424 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.311396 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-khx7t\"" Nov 02 21:34:56.362001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.361960 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5vwbc"] Nov 02 21:34:56.376041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.376014 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 21:34:56.376041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.376016 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 21:34:56.417481 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.417442 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnl8\" (UniqueName: \"kubernetes.io/projected/75a373ba-87a5-4ae2-89b2-1825e60e075b-kube-api-access-qsnl8\") pod \"downloads-6dd789c8b7-5vwbc\" (UID: \"75a373ba-87a5-4ae2-89b2-1825e60e075b\") " pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.417656 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.417503 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a373ba-87a5-4ae2-89b2-1825e60e075b-tmp\") pod \"downloads-6dd789c8b7-5vwbc\" (UID: \"75a373ba-87a5-4ae2-89b2-1825e60e075b\") " pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.518881 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.518807 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a373ba-87a5-4ae2-89b2-1825e60e075b-tmp\") pod \"downloads-6dd789c8b7-5vwbc\" (UID: \"75a373ba-87a5-4ae2-89b2-1825e60e075b\") " pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.518881 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.518859 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnl8\" (UniqueName: \"kubernetes.io/projected/75a373ba-87a5-4ae2-89b2-1825e60e075b-kube-api-access-qsnl8\") pod \"downloads-6dd789c8b7-5vwbc\" (UID: \"75a373ba-87a5-4ae2-89b2-1825e60e075b\") " pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.519319 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.519195 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a373ba-87a5-4ae2-89b2-1825e60e075b-tmp\") pod \"downloads-6dd789c8b7-5vwbc\" (UID: \"75a373ba-87a5-4ae2-89b2-1825e60e075b\") " pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.575379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.575349 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnl8\" (UniqueName: \"kubernetes.io/projected/75a373ba-87a5-4ae2-89b2-1825e60e075b-kube-api-access-qsnl8\") pod \"downloads-6dd789c8b7-5vwbc\" (UID: \"75a373ba-87a5-4ae2-89b2-1825e60e075b\") " pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.643235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.643197 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:56.866077 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.865995 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:34:56.970740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:56.970710 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbwsv" Nov 02 21:34:57.332065 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:57.332033 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5vwbc"] Nov 02 21:34:57.335360 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:34:57.335331 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a373ba_87a5_4ae2_89b2_1825e60e075b.slice/crio-2861b8dcdfbecc9383b4edc912c5c181b3346396b12ce0317cec5dad4b22e1d8 WatchSource:0}: Error finding container 2861b8dcdfbecc9383b4edc912c5c181b3346396b12ce0317cec5dad4b22e1d8: Status 404 returned error can't find the container with id 2861b8dcdfbecc9383b4edc912c5c181b3346396b12ce0317cec5dad4b22e1d8 Nov 02 21:34:57.374832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:57.374803 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5vwbc" event={"ID":"75a373ba-87a5-4ae2-89b2-1825e60e075b","Type":"ContainerStarted","Data":"2861b8dcdfbecc9383b4edc912c5c181b3346396b12ce0317cec5dad4b22e1d8"} Nov 02 21:34:58.028505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:58.028469 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:58.173256 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:58.173222 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:58.454961 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:58.454925 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:58.607579 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:58.607546 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:59.703442 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:59.703403 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:59.861625 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:34:59.861575 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:00.190553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:00.190483 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:00.340194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:00.340154 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:02.478641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:02.478603 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:02.649887 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:02.649801 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:05.262184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:05.262101 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:05.415523 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:05.415488 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:05.642991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:05.642907 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:35:05.642991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:05.642976 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:35:07.847524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.847485 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fddcc6565-gtn6l"] Nov 02 21:35:07.849608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.849587 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:07.951685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.951649 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 21:35:07.951685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.951650 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 21:35:07.951685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.951650 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 21:35:07.952013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.951756 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7rxng\"" Nov 02 21:35:07.952013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.951787 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 21:35:07.952013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.951846 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 21:35:07.953266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:07.953243 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fddcc6565-gtn6l"] Nov 02 21:35:08.011364 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011334 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9t8q\" (UniqueName: \"kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.011535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011373 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.011628 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011596 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.011705 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011665 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.011766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011730 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.011766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011756 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.011868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.011846 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112627 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112674 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112704 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112745 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9t8q\" (UniqueName: \"kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112776 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112822 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.112987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.112873 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.113524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.113495 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.113524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.113518 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.113678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.113524 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.113678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.113524 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.115515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.115496 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.118599 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.118578 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.169443 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.169407 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9t8q\" (UniqueName: \"kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q\") pod \"console-fddcc6565-gtn6l\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.462175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.462137 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:08.856229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:08.856119 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fddcc6565-gtn6l"] Nov 02 21:35:08.860100 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:35:08.860073 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986ef3c7_bd8c_4133_b4dd_d7bf259a71fc.slice/crio-c538c07d16681887916b8ab4cf9bdae437d346f12705540f81003ec8d21294fd WatchSource:0}: Error finding container c538c07d16681887916b8ab4cf9bdae437d346f12705540f81003ec8d21294fd: Status 404 returned error can't find the container with id c538c07d16681887916b8ab4cf9bdae437d346f12705540f81003ec8d21294fd Nov 02 21:35:09.422139 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:09.422088 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fddcc6565-gtn6l" event={"ID":"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc","Type":"ContainerStarted","Data":"c538c07d16681887916b8ab4cf9bdae437d346f12705540f81003ec8d21294fd"} Nov 02 21:35:10.217264 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:10.217224 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:10.363221 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:10.363181 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:12.436426 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:12.436387 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fddcc6565-gtn6l" event={"ID":"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc","Type":"ContainerStarted","Data":"257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d"} Nov 02 21:35:12.550408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:12.550261 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fddcc6565-gtn6l" podStartSLOduration=2.203787671 podStartE2EDuration="5.55024657s" totalImagesPullingTime="3.346458899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:12.550155935 +0000 UTC m=+51.511706281" watchObservedRunningTime="2026-11-02 21:35:12.55024657 +0000 UTC m=+51.511796872" Nov 02 21:35:13.259761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:13.259726 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.441727 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:13.441691 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5vwbc" event={"ID":"75a373ba-87a5-4ae2-89b2-1825e60e075b","Type":"ContainerStarted","Data":"7ac3eda5c10992ee5a7729edd0557b7d1be510fabc44a81d678386e034aca06e"} Nov 02 21:35:13.442441 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:13.442412 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:35:13.443963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:13.443936 4285 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-5vwbc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body= Nov 02 21:35:13.444076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:13.443994 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-5vwbc" podUID="75a373ba-87a5-4ae2-89b2-1825e60e075b" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" Nov 02 21:35:13.647818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:13.647712 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-5vwbc" podStartSLOduration=1.708851503 podStartE2EDuration="17.647691244s" totalImagesPullingTime="15.938839741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:13.604254435 +0000 UTC m=+52.565804755" watchObservedRunningTime="2026-11-02 21:35:13.647691244 +0000 UTC m=+52.609241562" Nov 02 21:35:14.464193 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:14.464161 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5vwbc" Nov 02 21:35:16.008980 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:16.008940 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:35:18.462657 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.462623 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:18.463169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.462687 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:18.468441 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.468412 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:18.570800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.570759 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d944f97-xfsw8"] Nov 02 21:35:18.605217 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.605184 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.622415 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.621692 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d944f97-xfsw8"] Nov 02 21:35:18.702725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.702688 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.702725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.702730 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.702965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.702754 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.702965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.702835 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.702965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.702896 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjck\" (UniqueName: \"kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.702965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.702921 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.703162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.703020 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.703162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.703054 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.718890 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.718821 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 21:35:18.803561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803524 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803567 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803600 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803633 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803667 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803688 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803730 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjck\" (UniqueName: \"kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.803802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.803754 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.804287 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.804259 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.804542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.804478 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.804542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.804478 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.805305 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.805279 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.805585 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.805563 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.806722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.806701 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.807326 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.807303 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.863603 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.863577 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjck\" (UniqueName: \"kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck\") pod \"console-69d944f97-xfsw8\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:18.936480 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:18.936442 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:19.337024 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:19.336988 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d944f97-xfsw8"] Nov 02 21:35:19.339896 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:35:19.339863 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaeaa79_8f20_44cf_bcfd_1080cff172a6.slice/crio-5aeea8993b8864ce98143837c8b72d3dc50c0199dc2212ff673c808c8d634983 WatchSource:0}: Error finding container 5aeea8993b8864ce98143837c8b72d3dc50c0199dc2212ff673c808c8d634983: Status 404 returned error can't find the container with id 5aeea8993b8864ce98143837c8b72d3dc50c0199dc2212ff673c808c8d634983 Nov 02 21:35:19.467571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:19.467530 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d944f97-xfsw8" event={"ID":"ccaeaa79-8f20-44cf-bcfd-1080cff172a6","Type":"ContainerStarted","Data":"d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9"} Nov 02 21:35:19.468013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:19.467596 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d944f97-xfsw8" event={"ID":"ccaeaa79-8f20-44cf-bcfd-1080cff172a6","Type":"ContainerStarted","Data":"5aeea8993b8864ce98143837c8b72d3dc50c0199dc2212ff673c808c8d634983"} Nov 02 21:35:19.472718 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:19.472656 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:19.581242 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:19.581178 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69d944f97-xfsw8" podStartSLOduration=1.581158151 podStartE2EDuration="1.581158151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:19.580619538 +0000 UTC m=+58.542169860" watchObservedRunningTime="2026-11-02 21:35:19.581158151 +0000 UTC m=+58.542708471" Nov 02 21:35:19.858905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:19.858864 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:20.005914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:20.005863 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:25.232656 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:25.232623 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:25.378641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:25.378588 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:25.648757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:25.648685 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:35:25.652510 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:25.652485 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dc98b96fb-mslq5" Nov 02 21:35:28.657963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:28.657931 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:28.805852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:28.805814 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:28.936635 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:28.936550 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:28.936635 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:28.936609 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:28.942688 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:28.942651 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:29.508193 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:29.508165 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:35:29.672048 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:29.672020 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fddcc6565-gtn6l"] Nov 02 21:35:30.872471 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:30.872437 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:31.029030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:31.028994 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:31.947186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:31.947147 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:32.224858 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:32.224764 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:34.083484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.083454 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:35:34.083914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.083754 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="alertmanager" containerID="cri-o://325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9" gracePeriod=120 Nov 02 21:35:34.083914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.083778 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy" containerID="cri-o://aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640" gracePeriod=120 Nov 02 21:35:34.083914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.083786 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="prom-label-proxy" containerID="cri-o://5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d" gracePeriod=120 Nov 02 21:35:34.083914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.083832 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-metric" containerID="cri-o://f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6" gracePeriod=120 Nov 02 21:35:34.084098 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.084026 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-web" containerID="cri-o://953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a" gracePeriod=120 Nov 02 21:35:34.084240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.084199 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="config-reloader" containerID="cri-o://1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060" gracePeriod=120 Nov 02 21:35:34.525234 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525202 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d" exitCode=0 Nov 02 21:35:34.525234 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525228 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6" exitCode=0 Nov 02 21:35:34.525234 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525235 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640" exitCode=0 Nov 02 21:35:34.525234 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525241 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060" exitCode=0 Nov 02 21:35:34.525513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525246 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9" exitCode=0 Nov 02 21:35:34.525513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525278 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d"} Nov 02 21:35:34.525513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525343 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6"} Nov 02 21:35:34.525513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525371 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640"} Nov 02 21:35:34.525513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525396 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060"} Nov 02 21:35:34.525513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:34.525422 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9"} Nov 02 21:35:35.329936 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.329912 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.437092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437011 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437054 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437080 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437100 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437147 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437177 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437201 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437224 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437255 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437296 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437337 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437369 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437416 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config\") pod \"d06b40f1-34b1-47a8-8194-ebda81966a22\" (UID: \"d06b40f1-34b1-47a8-8194-ebda81966a22\") " Nov 02 21:35:35.437820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437420 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:35.437820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437645 4285 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-main-db\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.437820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.437667 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:35.438379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.438086 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:35.440406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.440371 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:35.440652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.440614 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.440736 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.440688 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.440795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.440736 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.441207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.441175 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:35.441396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.441200 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "kube-api-access-jrz2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:35.441472 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.441455 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.441974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.441954 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.444437 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.444411 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.450976 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.450952 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config" pod "d06b40f1-34b1-47a8-8194-ebda81966a22" (UID: "d06b40f1-34b1-47a8-8194-ebda81966a22"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:35.531593 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.531560 4285 generic.go:356] "Generic (PLEG): container finished" podID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerID="953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a" exitCode=0 Nov 02 21:35:35.531754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.531620 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a"} Nov 02 21:35:35.531754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.531632 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.531754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.531685 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d06b40f1-34b1-47a8-8194-ebda81966a22","Type":"ContainerDied","Data":"24a56f3c3fdc302f9caadd9c30f4af8397be28444dbc5368c53138b3c0d82014"} Nov 02 21:35:35.531754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.531721 4285 scope.go:122] "RemoveContainer" containerID="5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d" Nov 02 21:35:35.538838 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538812 4285 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.538838 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538836 4285 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d06b40f1-34b1-47a8-8194-ebda81966a22-config-out\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538852 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrz2z\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-kube-api-access-jrz2z\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538865 4285 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538876 4285 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538885 4285 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-cluster-tls-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538893 4285 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538902 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-metrics-client-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538910 4285 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d06b40f1-34b1-47a8-8194-ebda81966a22-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538918 4285 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-secret-alertmanager-main-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538928 4285 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d06b40f1-34b1-47a8-8194-ebda81966a22-web-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.538936 4285 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d06b40f1-34b1-47a8-8194-ebda81966a22-tls-assets\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:35.539821 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.539804 4285 scope.go:122] "RemoveContainer" containerID="f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6" Nov 02 21:35:35.546443 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.546423 4285 scope.go:122] "RemoveContainer" containerID="aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640" Nov 02 21:35:35.553581 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.553563 4285 scope.go:122] "RemoveContainer" containerID="953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a" Nov 02 21:35:35.560102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.560081 4285 scope.go:122] "RemoveContainer" containerID="1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060" Nov 02 21:35:35.566979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.566959 4285 scope.go:122] "RemoveContainer" containerID="325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9" Nov 02 21:35:35.573589 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.573573 4285 scope.go:122] "RemoveContainer" containerID="fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c" Nov 02 21:35:35.580109 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.580092 4285 scope.go:122] "RemoveContainer" containerID="5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d" Nov 02 21:35:35.580488 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.580468 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d\": container with ID starting with 5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d not found: ID does not exist" containerID="5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d" Nov 02 21:35:35.580547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.580497 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d"} err="failed to get container status \"5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d\": rpc error: code = NotFound desc = could not find container \"5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d\": container with ID starting with 5602a7987d58e3a16baddb428ca1a872a5df0d6e51016f1aa19484603572226d not found: ID does not exist" Nov 02 21:35:35.580547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.580529 4285 scope.go:122] "RemoveContainer" containerID="f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6" Nov 02 21:35:35.580774 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.580759 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6\": container with ID starting with f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6 not found: ID does not exist" containerID="f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6" Nov 02 21:35:35.580809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.580781 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6"} err="failed to get container status \"f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6\": rpc error: code = NotFound desc = could not find container \"f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6\": container with ID starting with f08e80c2c7af8acd9670aa91a54b8c0d8790fffba3d8c526b8da22503aa3b4e6 not found: ID does not exist" Nov 02 21:35:35.580809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.580794 4285 scope.go:122] "RemoveContainer" containerID="aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640" Nov 02 21:35:35.581011 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.580990 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640\": container with ID starting with aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640 not found: ID does not exist" containerID="aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640" Nov 02 21:35:35.581093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581015 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640"} err="failed to get container status \"aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640\": rpc error: code = NotFound desc = could not find container \"aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640\": container with ID starting with aedc23072ce09f1abc7ad6017ac48e01d24087bc51c143d2a83df53cc9425640 not found: ID does not exist" Nov 02 21:35:35.581093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581031 4285 scope.go:122] "RemoveContainer" containerID="953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a" Nov 02 21:35:35.581313 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.581298 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a\": container with ID starting with 953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a not found: ID does not exist" containerID="953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a" Nov 02 21:35:35.581357 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581316 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a"} err="failed to get container status \"953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a\": rpc error: code = NotFound desc = could not find container \"953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a\": container with ID starting with 953714530d1e5839cf402f8a5023cc06d54799bee9ed9128f058b2a0161dd23a not found: ID does not exist" Nov 02 21:35:35.581357 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581329 4285 scope.go:122] "RemoveContainer" containerID="1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060" Nov 02 21:35:35.581563 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.581549 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060\": container with ID starting with 1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060 not found: ID does not exist" containerID="1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060" Nov 02 21:35:35.581607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581564 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060"} err="failed to get container status \"1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060\": rpc error: code = NotFound desc = could not find container \"1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060\": container with ID starting with 1f0e1e5d805f4ef18835164ce8a9aaae76e5e7e4ef495a04f5324b4ab4204060 not found: ID does not exist" Nov 02 21:35:35.581607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581574 4285 scope.go:122] "RemoveContainer" containerID="325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9" Nov 02 21:35:35.581809 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.581794 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9\": container with ID starting with 325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9 not found: ID does not exist" containerID="325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9" Nov 02 21:35:35.581856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581813 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9"} err="failed to get container status \"325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9\": rpc error: code = NotFound desc = could not find container \"325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9\": container with ID starting with 325a7d3ab048e1895039e6f048708070a090827f93546296c3c30b2aedd2b4c9 not found: ID does not exist" Nov 02 21:35:35.581856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.581826 4285 scope.go:122] "RemoveContainer" containerID="fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c" Nov 02 21:35:35.584194 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:35.584111 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c\": container with ID starting with fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c not found: ID does not exist" containerID="fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c" Nov 02 21:35:35.584194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.584165 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c"} err="failed to get container status \"fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c\": rpc error: code = NotFound desc = could not find container \"fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c\": container with ID starting with fdb53ab526c4dbd112c813247912ca5877d257bb5f2ddce58617881e5e4fa68c not found: ID does not exist" Nov 02 21:35:35.666064 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.666030 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:35:35.669043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.669018 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:35:35.693652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693627 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:35:35.693903 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693891 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="prom-label-proxy" Nov 02 21:35:35.693943 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693905 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="prom-label-proxy" Nov 02 21:35:35.693943 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693919 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy" Nov 02 21:35:35.693943 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693924 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy" Nov 02 21:35:35.693943 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693933 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-web" Nov 02 21:35:35.693943 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693938 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-web" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693950 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="alertmanager" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693954 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="alertmanager" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693960 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="config-reloader" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693964 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="config-reloader" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693973 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="init-config-reloader" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693977 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="init-config-reloader" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693983 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-metric" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.693988 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-metric" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.694034 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.694040 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-web" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.694047 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="alertmanager" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.694051 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="config-reloader" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.694056 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="kube-rbac-proxy-metric" Nov 02 21:35:35.694076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.694063 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" containerName="prom-label-proxy" Nov 02 21:35:35.697298 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.697280 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.740844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.740800 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c02fc47-54b8-4f5a-855b-d71972658fc5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.740844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.740843 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741056 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.740931 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741056 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.740993 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c02fc47-54b8-4f5a-855b-d71972658fc5-config-out\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741056 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741015 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741083 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741107 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c02fc47-54b8-4f5a-855b-d71972658fc5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741156 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741209 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c02fc47-54b8-4f5a-855b-d71972658fc5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741238 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct9t\" (UniqueName: \"kubernetes.io/projected/3c02fc47-54b8-4f5a-855b-d71972658fc5-kube-api-access-xct9t\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741273 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741315 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-web-config\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.741388 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.741341 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c02fc47-54b8-4f5a-855b-d71972658fc5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.750332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750222 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:35:35.750332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750231 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:35:35.750332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750296 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:35:35.750332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750295 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:35:35.750746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750384 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:35:35.750746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750518 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:35:35.750746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750566 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xjmjs\"" Nov 02 21:35:35.750746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750578 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:35:35.750746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750523 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:35:35.750986 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.750971 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:35:35.801069 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.801032 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:35:35.841981 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.841939 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.841981 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.841984 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c02fc47-54b8-4f5a-855b-d71972658fc5-config-out\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842227 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842006 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842227 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842181 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842227 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842219 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c02fc47-54b8-4f5a-855b-d71972658fc5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842248 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842284 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c02fc47-54b8-4f5a-855b-d71972658fc5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842309 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct9t\" (UniqueName: \"kubernetes.io/projected/3c02fc47-54b8-4f5a-855b-d71972658fc5-kube-api-access-xct9t\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842736 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842777 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-web-config\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842811 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c02fc47-54b8-4f5a-855b-d71972658fc5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.842885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842875 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c02fc47-54b8-4f5a-855b-d71972658fc5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.843163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.842904 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.843358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.843334 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3c02fc47-54b8-4f5a-855b-d71972658fc5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.843567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.843531 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c02fc47-54b8-4f5a-855b-d71972658fc5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.843825 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.843798 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c02fc47-54b8-4f5a-855b-d71972658fc5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.845170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.845142 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c02fc47-54b8-4f5a-855b-d71972658fc5-config-out\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.845261 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.845164 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.845332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.845286 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.845717 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.845697 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.845717 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.845712 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.845907 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.845882 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c02fc47-54b8-4f5a-855b-d71972658fc5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.846032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.846012 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-config-volume\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.846073 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.846049 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.846910 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.846895 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c02fc47-54b8-4f5a-855b-d71972658fc5-web-config\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:35.899355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:35.899318 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct9t\" (UniqueName: \"kubernetes.io/projected/3c02fc47-54b8-4f5a-855b-d71972658fc5-kube-api-access-xct9t\") pod \"alertmanager-main-0\" (UID: \"3c02fc47-54b8-4f5a-855b-d71972658fc5\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:36.013280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:36.013191 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:35:36.431957 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:36.431867 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:35:36.434717 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:35:36.434692 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c02fc47_54b8_4f5a_855b_d71972658fc5.slice/crio-1d8874515bec575f29582b8b02f08a3e08351c07f9b606aebb5e6f75f7514da2 WatchSource:0}: Error finding container 1d8874515bec575f29582b8b02f08a3e08351c07f9b606aebb5e6f75f7514da2: Status 404 returned error can't find the container with id 1d8874515bec575f29582b8b02f08a3e08351c07f9b606aebb5e6f75f7514da2 Nov 02 21:35:36.537558 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:36.537522 4285 generic.go:356] "Generic (PLEG): container finished" podID="3c02fc47-54b8-4f5a-855b-d71972658fc5" containerID="b0ce43a3f95bfc0ae7ad359c7ce4cd96fa359ebc1dac9eed1d657717dc57fded" exitCode=0 Nov 02 21:35:36.537703 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:36.537567 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerDied","Data":"b0ce43a3f95bfc0ae7ad359c7ce4cd96fa359ebc1dac9eed1d657717dc57fded"} Nov 02 21:35:36.537703 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:36.537623 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"1d8874515bec575f29582b8b02f08a3e08351c07f9b606aebb5e6f75f7514da2"} Nov 02 21:35:37.173317 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.173285 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06b40f1-34b1-47a8-8194-ebda81966a22" path="/var/lib/kubelet/pods/d06b40f1-34b1-47a8-8194-ebda81966a22/volumes" Nov 02 21:35:37.544844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.544809 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"3f346a3f75355ee49528760e8a1b0bc5591e3430ba74f4209c83b89926767cda"} Nov 02 21:35:37.545299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.544899 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"0ea683ddb4f5877aa6dfea5706857ebe4e3bace772d687b6fd7db19daef439c3"} Nov 02 21:35:37.545299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.544930 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"11ebe37f5b7fb3eb1bac6f52296d2f40b4661bab057d72fc7f336a312b0d9dd1"} Nov 02 21:35:37.545299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.544954 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"cef5625de5a11f2b83de7ea28bbaa03e808f347532aa75996e4be43c37bae5b2"} Nov 02 21:35:37.545299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.544978 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"bd4f8bd747e3f5b0e6f0d1e223643db1b57349327d9a3b2b1314e11fbe0a4e7f"} Nov 02 21:35:37.545299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.545001 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3c02fc47-54b8-4f5a-855b-d71972658fc5","Type":"ContainerStarted","Data":"4e058d961f81776b6e712d902fdf96b1a66712eb4005300b3e430ceca299a87a"} Nov 02 21:35:37.666817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.666768 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.65424055 podStartE2EDuration="2.66675606s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.51551ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:37.665691457 +0000 UTC m=+76.627241792" watchObservedRunningTime="2026-11-02 21:35:37.66675606 +0000 UTC m=+76.628306380" Nov 02 21:35:37.799280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.799208 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-86b48b8bb5-7j56s"] Nov 02 21:35:37.822536 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.822512 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860308 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860355 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-telemeter-client-tls\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860382 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqhh\" (UniqueName: \"kubernetes.io/projected/a96d8019-dc97-4a8e-9fca-be90d7795bcb-kube-api-access-gjqhh\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860416 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-metrics-client-ca\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860461 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-serving-certs-ca-bundle\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860490 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-secret-telemeter-client\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860531 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.860592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.860582 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-federate-client-tls\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.875161 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.875112 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 21:35:37.875304 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.875266 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 21:35:37.875304 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.875294 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 21:35:37.875424 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.875336 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 21:35:37.875424 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.875347 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 21:35:37.875582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.875562 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t95jp\"" Nov 02 21:35:37.925282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.925250 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 21:35:37.926326 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.926306 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86b48b8bb5-7j56s"] Nov 02 21:35:37.960942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.960912 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-metrics-client-ca\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.960957 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-serving-certs-ca-bundle\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.960989 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-secret-telemeter-client\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961030 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961181 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-federate-client-tls\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961226 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961330 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-telemeter-client-tls\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961375 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqhh\" (UniqueName: \"kubernetes.io/projected/a96d8019-dc97-4a8e-9fca-be90d7795bcb-kube-api-access-gjqhh\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961795 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-metrics-client-ca\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.961975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961813 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-serving-certs-ca-bundle\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.962094 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.961974 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a96d8019-dc97-4a8e-9fca-be90d7795bcb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.963647 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.963618 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.963739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.963675 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-federate-client-tls\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.963973 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.963958 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-telemeter-client-tls\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:37.963973 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:37.963965 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a96d8019-dc97-4a8e-9fca-be90d7795bcb-secret-telemeter-client\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:38.017316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:38.017285 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqhh\" (UniqueName: \"kubernetes.io/projected/a96d8019-dc97-4a8e-9fca-be90d7795bcb-kube-api-access-gjqhh\") pod \"telemeter-client-86b48b8bb5-7j56s\" (UID: \"a96d8019-dc97-4a8e-9fca-be90d7795bcb\") " pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:38.173594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:38.173520 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" Nov 02 21:35:38.668998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:38.668969 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86b48b8bb5-7j56s"] Nov 02 21:35:38.671648 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:35:38.671621 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96d8019_dc97_4a8e_9fca_be90d7795bcb.slice/crio-c082a69aa08861bc3770c7ff39dc815af6c2510b57070299de1e6991a58f200b WatchSource:0}: Error finding container c082a69aa08861bc3770c7ff39dc815af6c2510b57070299de1e6991a58f200b: Status 404 returned error can't find the container with id c082a69aa08861bc3770c7ff39dc815af6c2510b57070299de1e6991a58f200b Nov 02 21:35:39.553850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:39.553813 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" event={"ID":"a96d8019-dc97-4a8e-9fca-be90d7795bcb","Type":"ContainerStarted","Data":"c082a69aa08861bc3770c7ff39dc815af6c2510b57070299de1e6991a58f200b"} Nov 02 21:35:41.562474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:41.562434 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" event={"ID":"a96d8019-dc97-4a8e-9fca-be90d7795bcb","Type":"ContainerStarted","Data":"539dc9ef890ccadff503b6bddc435f02d6bb76e159af3604ccafcdbd766de630"} Nov 02 21:35:41.562849 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:41.562502 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" event={"ID":"a96d8019-dc97-4a8e-9fca-be90d7795bcb","Type":"ContainerStarted","Data":"74cb8f040a81e0c492d57b4ba25f1eb268902337857a83baa3c894c3f05119ae"} Nov 02 21:35:41.562849 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:41.562532 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" event={"ID":"a96d8019-dc97-4a8e-9fca-be90d7795bcb","Type":"ContainerStarted","Data":"ba2fb491daf839b8d7d1b9f6adf8bb8f382d6556e092e7ad8cae62653afba202"} Nov 02 21:35:42.190906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.190852 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-86b48b8bb5-7j56s" podStartSLOduration=2.67639843 podStartE2EDuration="5.190834301s" totalImagesPullingTime="2.514435871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:41.825864685 +0000 UTC m=+80.787415006" watchObservedRunningTime="2026-11-02 21:35:42.190834301 +0000 UTC m=+81.152384621" Nov 02 21:35:42.191352 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.191334 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b7df658f-4rjpb"] Nov 02 21:35:42.214140 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.214106 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.255368 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.255333 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7df658f-4rjpb"] Nov 02 21:35:42.298532 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298498 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.298704 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298540 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.298704 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298564 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.298820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298722 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.298820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298769 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.298908 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298883 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tr9\" (UniqueName: \"kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.298942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.298931 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.299018 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.299003 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.399832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.399798 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.399975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.399844 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tr9\" (UniqueName: \"kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400039 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.399962 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400039 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400008 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400051 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400115 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400318 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400183 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400318 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400224 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400494 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400473 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400670 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400862 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400834 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.400962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400939 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.401019 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.400966 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.402512 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.402486 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.402639 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.402620 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.455652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.455622 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tr9\" (UniqueName: \"kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9\") pod \"console-5b7df658f-4rjpb\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.482328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.482297 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:42.538092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.538057 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:42.630576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.630546 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:42.857236 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.857150 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:42.933606 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:42.933580 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7df658f-4rjpb"] Nov 02 21:35:42.936054 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:35:42.936025 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f65952_11e3_46ba_b62c_b6761e140317.slice/crio-beee9c90cdfe02f0070961567ab7241c8e8b17077e49d71260bbcb0e0ce29323 WatchSource:0}: Error finding container beee9c90cdfe02f0070961567ab7241c8e8b17077e49d71260bbcb0e0ce29323: Status 404 returned error can't find the container with id beee9c90cdfe02f0070961567ab7241c8e8b17077e49d71260bbcb0e0ce29323 Nov 02 21:35:43.008201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:43.008173 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:43.571863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:43.571823 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7df658f-4rjpb" event={"ID":"f2f65952-11e3-46ba-b62c-b6761e140317","Type":"ContainerStarted","Data":"7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120"} Nov 02 21:35:43.572022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:43.571873 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7df658f-4rjpb" event={"ID":"f2f65952-11e3-46ba-b62c-b6761e140317","Type":"ContainerStarted","Data":"beee9c90cdfe02f0070961567ab7241c8e8b17077e49d71260bbcb0e0ce29323"} Nov 02 21:35:43.685581 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:43.685521 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b7df658f-4rjpb" podStartSLOduration=1.685503503 podStartE2EDuration="1.685503503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:43.684331714 +0000 UTC m=+82.645882036" watchObservedRunningTime="2026-11-02 21:35:43.685503503 +0000 UTC m=+82.647053825" Nov 02 21:35:46.010894 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:46.010859 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:35:52.538902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:52.538857 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:52.539421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:52.539013 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:52.543660 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:52.543639 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:52.606835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:52.606810 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:35:52.800838 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:52.800764 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d944f97-xfsw8"] Nov 02 21:35:54.694527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:54.694496 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-fddcc6565-gtn6l" podUID="986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" containerName="console" containerID="cri-o://257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d" gracePeriod=15 Nov 02 21:35:54.970653 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:54.970600 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:55.108674 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108641 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q\" (UniqueName: \"kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.108832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108705 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.108832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108743 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.108832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108769 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.108993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108833 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp\" (UniqueName: \"kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.108993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108858 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.108993 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.108898 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config\") pod \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\" (UID: \"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc\") " Nov 02 21:35:55.109224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.109163 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:55.109283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.109224 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:55.109283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.109260 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:55.109283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.109267 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:55.110897 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.110873 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:55.110992 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.110936 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "kube-api-access-c9t8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:55.111045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.110997 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config" pod "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" (UID: "986ef3c7-bd8c-4133-b4dd-d7bf259a71fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:55.209840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209817 4285 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-tmp\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.209840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209838 4285 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-oauth-serving-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.209970 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209848 4285 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-oauth-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.209970 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209858 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9t8q\" (UniqueName: \"kubernetes.io/projected/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-kube-api-access-c9t8q\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.209970 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209868 4285 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.209970 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209877 4285 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.209970 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.209889 4285 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc-console-serving-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:35:55.222214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.222156 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:55.382697 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.382659 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:55.614739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.614659 4285 generic.go:356] "Generic (PLEG): container finished" podID="986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" containerID="257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d" exitCode=0 Nov 02 21:35:55.614739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.614714 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fddcc6565-gtn6l" Nov 02 21:35:55.614953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.614743 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fddcc6565-gtn6l" event={"ID":"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc","Type":"ContainerDied","Data":"257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d"} Nov 02 21:35:55.614953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.614791 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fddcc6565-gtn6l" event={"ID":"986ef3c7-bd8c-4133-b4dd-d7bf259a71fc","Type":"ContainerDied","Data":"c538c07d16681887916b8ab4cf9bdae437d346f12705540f81003ec8d21294fd"} Nov 02 21:35:55.614953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.614811 4285 scope.go:122] "RemoveContainer" containerID="257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d" Nov 02 21:35:55.622674 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.622648 4285 scope.go:122] "RemoveContainer" containerID="257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d" Nov 02 21:35:55.622914 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:35:55.622897 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d\": container with ID starting with 257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d not found: ID does not exist" containerID="257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d" Nov 02 21:35:55.622962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.622923 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d"} err="failed to get container status \"257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d\": rpc error: code = NotFound desc = could not find container \"257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d\": container with ID starting with 257c2f6ce3b901e78119af99feeb3f715b6c95306f9c00810d5936a23493702d not found: ID does not exist" Nov 02 21:35:55.775971 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.775942 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fddcc6565-gtn6l"] Nov 02 21:35:55.825540 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:55.825515 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fddcc6565-gtn6l"] Nov 02 21:35:57.174822 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:57.174788 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" path="/var/lib/kubelet/pods/986ef3c7-bd8c-4133-b4dd-d7bf259a71fc/volumes" Nov 02 21:35:59.688258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:35:59.688227 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:02.801889 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:02.801857 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:12.222741 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:12.222702 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:12.375809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:12.375773 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:16.005275 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:36:16.005241 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:36:16.576160 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:16.576110 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:17.819450 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:17.819413 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69d944f97-xfsw8" podUID="ccaeaa79-8f20-44cf-bcfd-1080cff172a6" containerName="console" containerID="cri-o://d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9" gracePeriod=15 Nov 02 21:36:18.067543 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.067521 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:36:18.108629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108547 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108598 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108623 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108654 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck\" (UniqueName: \"kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108672 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108706 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108754 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.108868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108791 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config\") pod \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\" (UID: \"ccaeaa79-8f20-44cf-bcfd-1080cff172a6\") " Nov 02 21:36:18.109150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.108914 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:18.109250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.109219 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:18.109310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.109225 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:18.109310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.109285 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:18.109447 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.109422 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:18.110747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.110717 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:18.111231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.111201 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:18.111336 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.111251 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck" pod "ccaeaa79-8f20-44cf-bcfd-1080cff172a6" (UID: "ccaeaa79-8f20-44cf-bcfd-1080cff172a6"). InnerVolumeSpecName "kube-api-access-9qjck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:18.209861 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209828 4285 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-oauth-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.209861 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209858 4285 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.209861 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209867 4285 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.210075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209875 4285 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-console-serving-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.210075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209884 4285 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-trusted-ca-bundle\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.210075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209893 4285 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-tmp\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.210075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209903 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjck\" (UniqueName: \"kubernetes.io/projected/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-kube-api-access-9qjck\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.210075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.209913 4285 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccaeaa79-8f20-44cf-bcfd-1080cff172a6-oauth-serving-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:18.545531 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545489 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74bb98d8b4-ctv7h"] Nov 02 21:36:18.545850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545831 4285 cpu_manager.go:414] "containerMap: removing container" podUID="986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" containerName="console" Nov 02 21:36:18.545928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545853 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" containerName="console" Nov 02 21:36:18.545928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545869 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ccaeaa79-8f20-44cf-bcfd-1080cff172a6" containerName="console" Nov 02 21:36:18.545928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545877 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccaeaa79-8f20-44cf-bcfd-1080cff172a6" containerName="console" Nov 02 21:36:18.546070 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545983 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="986ef3c7-bd8c-4133-b4dd-d7bf259a71fc" containerName="console" Nov 02 21:36:18.546070 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.545998 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccaeaa79-8f20-44cf-bcfd-1080cff172a6" containerName="console" Nov 02 21:36:18.547926 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.547905 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614073 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614032 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-oauth-config\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614073 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614075 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-oauth-serving-cert\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614177 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-service-ca\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614203 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmvg\" (UniqueName: \"kubernetes.io/projected/ccde9b8c-b94c-461a-be24-4fef051d90a5-kube-api-access-xjmvg\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614228 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-serving-cert\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614277 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-config\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614471 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614340 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-trusted-ca-bundle\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.614471 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.614397 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccde9b8c-b94c-461a-be24-4fef051d90a5-tmp\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.651019 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.650988 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bb98d8b4-ctv7h"] Nov 02 21:36:18.692825 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.692788 4285 generic.go:356] "Generic (PLEG): container finished" podID="ccaeaa79-8f20-44cf-bcfd-1080cff172a6" containerID="d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9" exitCode=0 Nov 02 21:36:18.692962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.692843 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d944f97-xfsw8" Nov 02 21:36:18.692962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.692875 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d944f97-xfsw8" event={"ID":"ccaeaa79-8f20-44cf-bcfd-1080cff172a6","Type":"ContainerDied","Data":"d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9"} Nov 02 21:36:18.692962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.692921 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d944f97-xfsw8" event={"ID":"ccaeaa79-8f20-44cf-bcfd-1080cff172a6","Type":"ContainerDied","Data":"5aeea8993b8864ce98143837c8b72d3dc50c0199dc2212ff673c808c8d634983"} Nov 02 21:36:18.692962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.692945 4285 scope.go:122] "RemoveContainer" containerID="d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9" Nov 02 21:36:18.701473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.701454 4285 scope.go:122] "RemoveContainer" containerID="d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9" Nov 02 21:36:18.701708 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:36:18.701691 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9\": container with ID starting with d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9 not found: ID does not exist" containerID="d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9" Nov 02 21:36:18.701757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.701719 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9"} err="failed to get container status \"d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9\": rpc error: code = NotFound desc = could not find container \"d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9\": container with ID starting with d78d61d2a33d97e3b344aa9d01b2bb6f5b757327d69f38e0a4b8e6d9c95eb6d9 not found: ID does not exist" Nov 02 21:36:18.715206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715179 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-serving-cert\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715215 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-config\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715231 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-trusted-ca-bundle\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715250 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccde9b8c-b94c-461a-be24-4fef051d90a5-tmp\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715300 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-oauth-config\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715324 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-oauth-serving-cert\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715363 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-service-ca\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715519 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715386 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmvg\" (UniqueName: \"kubernetes.io/projected/ccde9b8c-b94c-461a-be24-4fef051d90a5-kube-api-access-xjmvg\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715771 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715742 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccde9b8c-b94c-461a-be24-4fef051d90a5-tmp\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.715957 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.715938 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-config\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.716053 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.716029 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-oauth-serving-cert\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.716213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.716197 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-service-ca\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.716256 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.716216 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccde9b8c-b94c-461a-be24-4fef051d90a5-trusted-ca-bundle\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.717677 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.717656 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-serving-cert\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.717756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.717689 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccde9b8c-b94c-461a-be24-4fef051d90a5-console-oauth-config\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.771931 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.771902 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmvg\" (UniqueName: \"kubernetes.io/projected/ccde9b8c-b94c-461a-be24-4fef051d90a5-kube-api-access-xjmvg\") pod \"console-74bb98d8b4-ctv7h\" (UID: \"ccde9b8c-b94c-461a-be24-4fef051d90a5\") " pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:18.838200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.838135 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:18.853799 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.853768 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d944f97-xfsw8"] Nov 02 21:36:18.857185 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.857162 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d944f97-xfsw8"] Nov 02 21:36:18.858213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:18.858196 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:19.170724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.170648 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccaeaa79-8f20-44cf-bcfd-1080cff172a6" path="/var/lib/kubelet/pods/ccaeaa79-8f20-44cf-bcfd-1080cff172a6/volumes" Nov 02 21:36:19.206398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.206370 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:19.223646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.223625 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bb98d8b4-ctv7h"] Nov 02 21:36:19.226306 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:36:19.226279 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccde9b8c_b94c_461a_be24_4fef051d90a5.slice/crio-fbaf0e336af84a9eb65ffcd8b2b0c7c01a0548633cd2d4a0508baae4b1c54da8 WatchSource:0}: Error finding container fbaf0e336af84a9eb65ffcd8b2b0c7c01a0548633cd2d4a0508baae4b1c54da8: Status 404 returned error can't find the container with id fbaf0e336af84a9eb65ffcd8b2b0c7c01a0548633cd2d4a0508baae4b1c54da8 Nov 02 21:36:19.350464 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.350430 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:19.698712 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.698677 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bb98d8b4-ctv7h" event={"ID":"ccde9b8c-b94c-461a-be24-4fef051d90a5","Type":"ContainerStarted","Data":"8290a4e1c4a793bd400eb76930fb6a40eedb6419783af45c292278927d31ca79"} Nov 02 21:36:19.698880 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.698724 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bb98d8b4-ctv7h" event={"ID":"ccde9b8c-b94c-461a-be24-4fef051d90a5","Type":"ContainerStarted","Data":"fbaf0e336af84a9eb65ffcd8b2b0c7c01a0548633cd2d4a0508baae4b1c54da8"} Nov 02 21:36:19.812335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:19.812272 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74bb98d8b4-ctv7h" podStartSLOduration=1.812253164 podStartE2EDuration="1.812253164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:19.811313422 +0000 UTC m=+118.772863753" watchObservedRunningTime="2026-11-02 21:36:19.812253164 +0000 UTC m=+118.773803485" Nov 02 21:36:25.215893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:25.215856 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:25.371241 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:25.371207 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:28.858483 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:28.858449 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:28.858879 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:28.858527 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:28.863135 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:28.863095 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:29.736637 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:29.736610 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74bb98d8b4-ctv7h" Nov 02 21:36:29.899920 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:29.899887 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b7df658f-4rjpb"] Nov 02 21:36:46.004962 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:36:46.004879 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:36:54.919120 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:54.919079 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b7df658f-4rjpb" podUID="f2f65952-11e3-46ba-b62c-b6761e140317" containerName="console" containerID="cri-o://7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120" gracePeriod=15 Nov 02 21:36:55.159860 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.159838 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:36:55.314918 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.314893 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315090 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.314924 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315090 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.314948 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315090 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.314987 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315176 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315254 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:55.315368 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315276 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9\" (UniqueName: \"kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315368 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315317 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315478 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315371 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config\") pod \"f2f65952-11e3-46ba-b62c-b6761e140317\" (UID: \"f2f65952-11e3-46ba-b62c-b6761e140317\") " Nov 02 21:36:55.315478 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315391 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:55.315478 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315450 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:55.315682 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315515 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:55.315720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315689 4285 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-oauth-serving-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.315720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315710 4285 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f65952-11e3-46ba-b62c-b6761e140317-tmp\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.315797 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315724 4285 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.315797 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315738 4285 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-trusted-ca-bundle\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.315889 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.315810 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:55.317212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.317186 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:55.317398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.317375 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:55.317465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.317385 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9" pod "f2f65952-11e3-46ba-b62c-b6761e140317" (UID: "f2f65952-11e3-46ba-b62c-b6761e140317"). InnerVolumeSpecName "kube-api-access-f9tr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:55.416901 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.416874 4285 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2f65952-11e3-46ba-b62c-b6761e140317-console-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.416901 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.416899 4285 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-serving-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.417057 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.416909 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9tr9\" (UniqueName: \"kubernetes.io/projected/f2f65952-11e3-46ba-b62c-b6761e140317-kube-api-access-f9tr9\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.417057 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.416918 4285 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2f65952-11e3-46ba-b62c-b6761e140317-console-oauth-config\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:36:55.818702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.818665 4285 generic.go:356] "Generic (PLEG): container finished" podID="f2f65952-11e3-46ba-b62c-b6761e140317" containerID="7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120" exitCode=0 Nov 02 21:36:55.818867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.818708 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7df658f-4rjpb" Nov 02 21:36:55.818867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.818723 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7df658f-4rjpb" event={"ID":"f2f65952-11e3-46ba-b62c-b6761e140317","Type":"ContainerDied","Data":"7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120"} Nov 02 21:36:55.818867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.818770 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7df658f-4rjpb" event={"ID":"f2f65952-11e3-46ba-b62c-b6761e140317","Type":"ContainerDied","Data":"beee9c90cdfe02f0070961567ab7241c8e8b17077e49d71260bbcb0e0ce29323"} Nov 02 21:36:55.818867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.818791 4285 scope.go:122] "RemoveContainer" containerID="7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120" Nov 02 21:36:55.829168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.829146 4285 scope.go:122] "RemoveContainer" containerID="7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120" Nov 02 21:36:55.829604 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:36:55.829585 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120\": container with ID starting with 7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120 not found: ID does not exist" containerID="7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120" Nov 02 21:36:55.829694 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.829616 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120"} err="failed to get container status \"7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120\": rpc error: code = NotFound desc = could not find container \"7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120\": container with ID starting with 7d35d575b47ad0ef326b0cea679c454633789a83154d23678d456e80423f7120 not found: ID does not exist" Nov 02 21:36:55.979903 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.979869 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b7df658f-4rjpb"] Nov 02 21:36:55.983164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:55.983144 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b7df658f-4rjpb"] Nov 02 21:36:57.170077 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:36:57.170043 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f65952-11e3-46ba-b62c-b6761e140317" path="/var/lib/kubelet/pods/f2f65952-11e3-46ba-b62c-b6761e140317/volumes" Nov 02 21:37:03.128848 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:37:03.128813 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:03.409511 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:37:03.409431 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:16.004241 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:37:16.004203 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:37:46.004730 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:37:46.004686 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:37:56.637452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:37:56.637422 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:56.781524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:37:56.781497 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:03.463220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:03.463191 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:16.004272 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:38:16.004216 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:38:22.102152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.102088 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:22.900878 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.900844 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm"] Nov 02 21:38:22.901147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.901112 4285 cpu_manager.go:414] "containerMap: removing container" podUID="f2f65952-11e3-46ba-b62c-b6761e140317" containerName="console" Nov 02 21:38:22.901197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.901150 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f65952-11e3-46ba-b62c-b6761e140317" containerName="console" Nov 02 21:38:22.901230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.901207 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f65952-11e3-46ba-b62c-b6761e140317" containerName="console" Nov 02 21:38:22.904079 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.904061 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:22.925156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.925113 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z"] Nov 02 21:38:22.928273 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.928255 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm"] Nov 02 21:38:22.928375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.928338 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:22.962253 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.962236 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-52j4l\"" Nov 02 21:38:22.962253 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.962242 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 21:38:22.973705 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.973678 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 21:38:22.992831 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:22.992806 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z"] Nov 02 21:38:23.029661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.029636 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnmfj\" (UniqueName: \"kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.029774 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.029675 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.029774 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.029698 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.029850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.029794 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttr8\" (UniqueName: \"kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.029894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.029854 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.029962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.029948 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.063138 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.063107 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-njfrv\"" Nov 02 21:38:23.130403 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130378 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnmfj\" (UniqueName: \"kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.130747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130406 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.130747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130421 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.130747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130444 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttr8\" (UniqueName: \"kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.130747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130483 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.130747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130512 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.130919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130800 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.130919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130861 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.130919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130878 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.130919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.130906 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.192916 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.192896 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnmfj\" (UniqueName: \"kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.193030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.193008 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttr8\" (UniqueName: \"kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.220758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.220737 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:23.242846 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.242814 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:23.256072 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.256046 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd"] Nov 02 21:38:23.260190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.259263 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.313980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.313870 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 21:38:23.313980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.313910 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:38:23.313980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.313945 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:38:23.319112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.319091 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 21:38:23.319112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.319111 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z7n66\"" Nov 02 21:38:23.319112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.319141 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 21:38:23.319112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.319115 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 21:38:23.319419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.319282 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd"] Nov 02 21:38:23.433883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.433844 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.434038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.433896 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.434038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.433962 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.434038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.434011 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5vk\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.434038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.434035 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.434213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.434157 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.434213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.434183 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535149 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5vk\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535191 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535405 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535245 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535405 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535275 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535405 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535309 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535405 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535348 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535405 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535372 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.535904 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.535883 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.537500 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.537477 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.537590 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.537538 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.537757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.537734 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.537863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.537849 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.591175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.591151 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.591372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.591354 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5vk\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk\") pod \"istiod-openshift-gateway-7876bfcb74-h99jd\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:23.618118 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.618100 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm"] Nov 02 21:38:23.620192 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:38:23.620171 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7412c211_23a6_4de9_839e_b6ea3e0c5438.slice/crio-51b3e0026d42c4ac354a4d67324b7444d87fcb37e444a004da913ab0a7b4e4ef WatchSource:0}: Error finding container 51b3e0026d42c4ac354a4d67324b7444d87fcb37e444a004da913ab0a7b4e4ef: Status 404 returned error can't find the container with id 51b3e0026d42c4ac354a4d67324b7444d87fcb37e444a004da913ab0a7b4e4ef Nov 02 21:38:23.666253 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.666232 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z"] Nov 02 21:38:23.668111 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:38:23.668086 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946ee367_494c_4735_a1e2_1f1e2cf3a7cb.slice/crio-228f69e2dab3bf788db0d5442cbba0a5b9f554cdbb3f9ced84a0b8e712e6ea9a WatchSource:0}: Error finding container 228f69e2dab3bf788db0d5442cbba0a5b9f554cdbb3f9ced84a0b8e712e6ea9a: Status 404 returned error can't find the container with id 228f69e2dab3bf788db0d5442cbba0a5b9f554cdbb3f9ced84a0b8e712e6ea9a Nov 02 21:38:23.875201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:23.875120 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:24.117329 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:24.117282 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" event={"ID":"7412c211-23a6-4de9-839e-b6ea3e0c5438","Type":"ContainerStarted","Data":"51b3e0026d42c4ac354a4d67324b7444d87fcb37e444a004da913ab0a7b4e4ef"} Nov 02 21:38:24.118334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:24.118308 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" event={"ID":"946ee367-494c-4735-a1e2-1f1e2cf3a7cb","Type":"ContainerStarted","Data":"228f69e2dab3bf788db0d5442cbba0a5b9f554cdbb3f9ced84a0b8e712e6ea9a"} Nov 02 21:38:24.241649 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:24.241379 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd"] Nov 02 21:38:24.243839 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:38:24.243810 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d307a43_fd7d_4b9a_aebe_7c00cb169b2e.slice/crio-316cd476e7fe2dc9f4ac92dd121a4958722405a3d1c5d8e50f45403e0c470629 WatchSource:0}: Error finding container 316cd476e7fe2dc9f4ac92dd121a4958722405a3d1c5d8e50f45403e0c470629: Status 404 returned error can't find the container with id 316cd476e7fe2dc9f4ac92dd121a4958722405a3d1c5d8e50f45403e0c470629 Nov 02 21:38:24.251482 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:24.251461 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:25.124854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:25.124816 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" event={"ID":"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e","Type":"ContainerStarted","Data":"316cd476e7fe2dc9f4ac92dd121a4958722405a3d1c5d8e50f45403e0c470629"} Nov 02 21:38:28.485949 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:28.485916 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:38:28.486497 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:28.486456 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:38:29.148350 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:29.148311 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" event={"ID":"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e","Type":"ContainerStarted","Data":"f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea"} Nov 02 21:38:29.148598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:29.148567 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:29.150627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:29.150598 4285 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-h99jd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:38:29.150728 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:29.150654 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:38:29.356352 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:29.355383 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" podStartSLOduration=2.116009663 podStartE2EDuration="6.355366776s" totalImagesPullingTime="4.239357113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:29.311475142 +0000 UTC m=+248.273025461" watchObservedRunningTime="2026-11-02 21:38:29.355366776 +0000 UTC m=+248.316917099" Nov 02 21:38:30.154093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:30.154060 4285 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-h99jd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:38:30.154494 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:30.154120 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:38:30.168890 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:30.168851 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:30.324832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:30.324799 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:31.117766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.117725 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:31.157759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.157726 4285 generic.go:356] "Generic (PLEG): container finished" podID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerID="da7412fba71046ee0f7ac91249449f5603d836bd5efe5906ec6d66b0fc6f0f16" exitCode=0 Nov 02 21:38:31.158142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.157816 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" event={"ID":"7412c211-23a6-4de9-839e-b6ea3e0c5438","Type":"ContainerDied","Data":"da7412fba71046ee0f7ac91249449f5603d836bd5efe5906ec6d66b0fc6f0f16"} Nov 02 21:38:31.159393 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.159373 4285 generic.go:356] "Generic (PLEG): container finished" podID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerID="3fc78627039432c7aa6649e1a7b1e0d9406a09d45fdbd849fde18ff4540a92f7" exitCode=0 Nov 02 21:38:31.159460 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.159423 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" event={"ID":"946ee367-494c-4735-a1e2-1f1e2cf3a7cb","Type":"ContainerDied","Data":"3fc78627039432c7aa6649e1a7b1e0d9406a09d45fdbd849fde18ff4540a92f7"} Nov 02 21:38:31.404929 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.404893 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:31.555348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:31.555316 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:33.153953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:33.153925 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:38:33.167883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:33.167856 4285 generic.go:356] "Generic (PLEG): container finished" podID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerID="e900defccd282b71446d7a8ef4726f950dd20d5ff0ed5eef7366a93b08c72933" exitCode=0 Nov 02 21:38:33.170691 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:33.170667 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" event={"ID":"7412c211-23a6-4de9-839e-b6ea3e0c5438","Type":"ContainerDied","Data":"e900defccd282b71446d7a8ef4726f950dd20d5ff0ed5eef7366a93b08c72933"} Nov 02 21:38:35.413398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:35.413356 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:36.181844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:36.181751 4285 generic.go:356] "Generic (PLEG): container finished" podID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerID="add66e3a8ed3e9350d42102e6e4fb94370154bb7c2cf0e48d1723eda98018a8b" exitCode=0 Nov 02 21:38:36.181844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:36.181799 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" event={"ID":"946ee367-494c-4735-a1e2-1f1e2cf3a7cb","Type":"ContainerDied","Data":"add66e3a8ed3e9350d42102e6e4fb94370154bb7c2cf0e48d1723eda98018a8b"} Nov 02 21:38:40.200338 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:40.200299 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" event={"ID":"7412c211-23a6-4de9-839e-b6ea3e0c5438","Type":"ContainerStarted","Data":"0af22545f0fd5d455d7a3329cf0a147906ead3630dbd0703de2d8e4ec7bc70f8"} Nov 02 21:38:40.316008 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:40.315960 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" podStartSLOduration=3.6476409260000002 podStartE2EDuration="18.315941705s" totalImagesPullingTime="14.668300779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:40.315287089 +0000 UTC m=+259.276837409" watchObservedRunningTime="2026-11-02 21:38:40.315941705 +0000 UTC m=+259.277492019" Nov 02 21:38:41.205893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:41.205861 4285 generic.go:356] "Generic (PLEG): container finished" podID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerID="0af22545f0fd5d455d7a3329cf0a147906ead3630dbd0703de2d8e4ec7bc70f8" exitCode=0 Nov 02 21:38:41.208275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:41.205972 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" event={"ID":"7412c211-23a6-4de9-839e-b6ea3e0c5438","Type":"ContainerDied","Data":"0af22545f0fd5d455d7a3329cf0a147906ead3630dbd0703de2d8e4ec7bc70f8"} Nov 02 21:38:41.208275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:41.208055 4285 generic.go:356] "Generic (PLEG): container finished" podID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerID="a024a77815945da1995189a821bd741dbc82cb1146cb6aa32d659a5b7402c38c" exitCode=0 Nov 02 21:38:41.208275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:41.208094 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" event={"ID":"946ee367-494c-4735-a1e2-1f1e2cf3a7cb","Type":"ContainerDied","Data":"a024a77815945da1995189a821bd741dbc82cb1146cb6aa32d659a5b7402c38c"} Nov 02 21:38:42.324634 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.324610 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:42.359614 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.359594 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:42.399454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399435 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8\" (UniqueName: \"kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8\") pod \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " Nov 02 21:38:42.399554 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399468 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle\") pod \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " Nov 02 21:38:42.399554 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399485 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util\") pod \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\" (UID: \"946ee367-494c-4735-a1e2-1f1e2cf3a7cb\") " Nov 02 21:38:42.399631 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399608 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util\") pod \"7412c211-23a6-4de9-839e-b6ea3e0c5438\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " Nov 02 21:38:42.399690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399674 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj\" (UniqueName: \"kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj\") pod \"7412c211-23a6-4de9-839e-b6ea3e0c5438\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " Nov 02 21:38:42.399731 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399711 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle\") pod \"7412c211-23a6-4de9-839e-b6ea3e0c5438\" (UID: \"7412c211-23a6-4de9-839e-b6ea3e0c5438\") " Nov 02 21:38:42.399986 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.399942 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle" pod "946ee367-494c-4735-a1e2-1f1e2cf3a7cb" (UID: "946ee367-494c-4735-a1e2-1f1e2cf3a7cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:42.400324 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.400291 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle" pod "7412c211-23a6-4de9-839e-b6ea3e0c5438" (UID: "7412c211-23a6-4de9-839e-b6ea3e0c5438"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:42.401719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.401690 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj" pod "7412c211-23a6-4de9-839e-b6ea3e0c5438" (UID: "7412c211-23a6-4de9-839e-b6ea3e0c5438"). InnerVolumeSpecName "kube-api-access-cnmfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:42.401861 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.401839 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8" pod "946ee367-494c-4735-a1e2-1f1e2cf3a7cb" (UID: "946ee367-494c-4735-a1e2-1f1e2cf3a7cb"). InnerVolumeSpecName "kube-api-access-5ttr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:42.403629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.403610 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util" pod "946ee367-494c-4735-a1e2-1f1e2cf3a7cb" (UID: "946ee367-494c-4735-a1e2-1f1e2cf3a7cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:42.404313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.404294 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util" pod "7412c211-23a6-4de9-839e-b6ea3e0c5438" (UID: "7412c211-23a6-4de9-839e-b6ea3e0c5438"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:42.500975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.500914 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnmfj\" (UniqueName: \"kubernetes.io/projected/7412c211-23a6-4de9-839e-b6ea3e0c5438-kube-api-access-cnmfj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:42.500975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.500936 4285 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-bundle\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:42.500975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.500945 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttr8\" (UniqueName: \"kubernetes.io/projected/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-kube-api-access-5ttr8\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:42.500975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.500953 4285 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-bundle\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:42.500975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.500961 4285 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/946ee367-494c-4735-a1e2-1f1e2cf3a7cb-util\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:42.500975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:42.500968 4285 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7412c211-23a6-4de9-839e-b6ea3e0c5438-util\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:38:43.216483 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:43.216462 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" Nov 02 21:38:43.216621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:43.216449 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rqx7z" event={"ID":"946ee367-494c-4735-a1e2-1f1e2cf3a7cb","Type":"ContainerDied","Data":"228f69e2dab3bf788db0d5442cbba0a5b9f554cdbb3f9ced84a0b8e712e6ea9a"} Nov 02 21:38:43.216621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:43.216606 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228f69e2dab3bf788db0d5442cbba0a5b9f554cdbb3f9ced84a0b8e712e6ea9a" Nov 02 21:38:43.218462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:43.218444 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" Nov 02 21:38:43.218576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:43.218440 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71f25bm" event={"ID":"7412c211-23a6-4de9-839e-b6ea3e0c5438","Type":"ContainerDied","Data":"51b3e0026d42c4ac354a4d67324b7444d87fcb37e444a004da913ab0a7b4e4ef"} Nov 02 21:38:43.218576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:43.218568 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b3e0026d42c4ac354a4d67324b7444d87fcb37e444a004da913ab0a7b4e4ef" Nov 02 21:38:45.800662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:45.800623 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:46.003363 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:38:46.003330 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:38:57.300777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:57.300730 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:58.687280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:38:58.687252 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:00.904184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:00.904149 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:01.067589 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:01.067564 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:02.920966 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:02.920931 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:03.083780 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:03.083751 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:03.798464 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:03.798431 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:07.509366 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:07.509326 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:07.762471 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:07.762394 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:08.592466 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.592438 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:08.796875 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.796838 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs"] Nov 02 21:39:08.797215 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797197 4285 cpu_manager.go:414] "containerMap: removing container" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="util" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797216 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="util" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797232 4285 cpu_manager.go:414] "containerMap: removing container" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="pull" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797238 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="pull" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797251 4285 cpu_manager.go:414] "containerMap: removing container" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="pull" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797259 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="pull" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797281 4285 cpu_manager.go:414] "containerMap: removing container" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="util" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797288 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="util" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797303 4285 cpu_manager.go:414] "containerMap: removing container" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="extract" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797309 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="extract" Nov 02 21:39:08.797315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797319 4285 cpu_manager.go:414] "containerMap: removing container" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="extract" Nov 02 21:39:08.797770 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797326 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="extract" Nov 02 21:39:08.797770 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797410 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="7412c211-23a6-4de9-839e-b6ea3e0c5438" containerName="extract" Nov 02 21:39:08.797770 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.797421 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="946ee367-494c-4735-a1e2-1f1e2cf3a7cb" containerName="extract" Nov 02 21:39:08.799245 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.799224 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.813791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.813769 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:08.893944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.893869 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/136139ca-b681-464a-9f91-48060afd3478-tmp\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.893944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.893913 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/136139ca-b681-464a-9f91-48060afd3478-manifests\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.894173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.893996 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.894173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.894086 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.894173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.894162 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbq2j\" (UniqueName: \"kubernetes.io/projected/136139ca-b681-464a-9f91-48060afd3478-kube-api-access-zbq2j\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.894315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.894192 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.930863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.930839 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 21:39:08.931004 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.930880 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 21:39:08.931004 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.930954 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 21:39:08.931004 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.930998 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 21:39:08.931201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.931004 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cf4br\"" Nov 02 21:39:08.931201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.931017 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 21:39:08.935020 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.934998 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs"] Nov 02 21:39:08.994652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.994619 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/136139ca-b681-464a-9f91-48060afd3478-tmp\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.994652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.994658 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/136139ca-b681-464a-9f91-48060afd3478-manifests\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.994893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.994682 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.994893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.994706 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.994893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.994726 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbq2j\" (UniqueName: \"kubernetes.io/projected/136139ca-b681-464a-9f91-48060afd3478-kube-api-access-zbq2j\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.994893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.994741 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.995097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.995078 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/136139ca-b681-464a-9f91-48060afd3478-manifests\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.997091 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.997070 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/136139ca-b681-464a-9f91-48060afd3478-tmp\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.997334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.997314 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.997417 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.997396 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:08.997488 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:08.997469 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/136139ca-b681-464a-9f91-48060afd3478-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:09.002076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:09.002058 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:09.052418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:09.052387 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbq2j\" (UniqueName: \"kubernetes.io/projected/136139ca-b681-464a-9f91-48060afd3478-kube-api-access-zbq2j\") pod \"opendatahub-operator-controller-manager-67f87dc46d-z6lbs\" (UID: \"136139ca-b681-464a-9f91-48060afd3478\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:09.111660 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:09.111632 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:09.178106 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:09.178077 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:09.589335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:09.589311 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs"] Nov 02 21:39:09.591917 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:39:09.591887 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136139ca_b681_464a_9f91_48060afd3478.slice/crio-f78105fc05228ea2ebf79f393f2541f743fc1a2fd75645a136d0347b95e59ad9 WatchSource:0}: Error finding container f78105fc05228ea2ebf79f393f2541f743fc1a2fd75645a136d0347b95e59ad9: Status 404 returned error can't find the container with id f78105fc05228ea2ebf79f393f2541f743fc1a2fd75645a136d0347b95e59ad9 Nov 02 21:39:10.310700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:10.310665 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" event={"ID":"136139ca-b681-464a-9f91-48060afd3478","Type":"ContainerStarted","Data":"f78105fc05228ea2ebf79f393f2541f743fc1a2fd75645a136d0347b95e59ad9"} Nov 02 21:39:11.649398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:11.649365 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:12.101809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:12.101782 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:12.258633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:12.258537 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:13.325081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:13.325041 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" event={"ID":"136139ca-b681-464a-9f91-48060afd3478","Type":"ContainerStarted","Data":"ecc1aa4c256f7cbbfa597653ae72834cbf287fae7010d596d15f95fc003e4f31"} Nov 02 21:39:14.330293 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:14.330256 4285 generic.go:356] "Generic (PLEG): container finished" podID="136139ca-b681-464a-9f91-48060afd3478" containerID="ecc1aa4c256f7cbbfa597653ae72834cbf287fae7010d596d15f95fc003e4f31" exitCode=0 Nov 02 21:39:14.330690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:14.330344 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" event={"ID":"136139ca-b681-464a-9f91-48060afd3478","Type":"ContainerDied","Data":"ecc1aa4c256f7cbbfa597653ae72834cbf287fae7010d596d15f95fc003e4f31"} Nov 02 21:39:15.105792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:15.105756 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:15.335329 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:15.335294 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" event={"ID":"136139ca-b681-464a-9f91-48060afd3478","Type":"ContainerStarted","Data":"af68f579928b2017ebfe787c7e3284ceb8f018bc0a630afb59c9da6ccab2802a"} Nov 02 21:39:15.335690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:15.335404 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:15.467753 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:15.467707 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" podStartSLOduration=2.530207122 podStartE2EDuration="7.467693187s" totalImagesPullingTime="4.450958285s" totalInitContainerRuntime="486.52778ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:15.465777805 +0000 UTC m=+294.427328165" watchObservedRunningTime="2026-11-02 21:39:15.467693187 +0000 UTC m=+294.429243505" Nov 02 21:39:16.003761 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:39:16.003726 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:39:18.572586 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:18.572554 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:18.749999 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:18.749969 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:19.136165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:19.136121 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:19.907272 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:19.907238 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:20.701384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:20.701342 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:25.548989 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:25.548955 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:26.340496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:26.340469 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f87dc46d-z6lbs" Nov 02 21:39:27.935947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:27.935913 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.567342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:28.567274 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.773254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:28.773213 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.938282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:28.938200 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:30.905305 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:30.905264 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:31.530811 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:31.530766 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:37.808468 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:37.808372 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:38.166295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:38.166174 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:39.101475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:39.101442 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:39.274916 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:39.274872 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:39.628297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:39.628268 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:39.800656 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:39.800618 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:40.252546 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:40.252511 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:40.408867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:40.408837 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:40.801996 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:40.801961 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:41.130477 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:41.130400 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:41.973646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:41.973615 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:42.224967 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.224897 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx"] Nov 02 21:39:42.228091 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.228077 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.239321 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.239297 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:42.263065 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.263040 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8d0ba1-1b77-4850-b6ad-960a92533271-tls-certs\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.263065 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.263066 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e8d0ba1-1b77-4850-b6ad-960a92533271-tmp\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.263224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.263185 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mxq\" (UniqueName: \"kubernetes.io/projected/4e8d0ba1-1b77-4850-b6ad-960a92533271-kube-api-access-g8mxq\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.279882 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.279860 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 21:39:42.279976 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.279888 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 21:39:42.329401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.329369 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xgxvs\"" Nov 02 21:39:42.332007 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.331978 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx"] Nov 02 21:39:42.363849 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.363809 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8d0ba1-1b77-4850-b6ad-960a92533271-tls-certs\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.363978 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.363852 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e8d0ba1-1b77-4850-b6ad-960a92533271-tmp\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.363978 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.363930 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mxq\" (UniqueName: \"kubernetes.io/projected/4e8d0ba1-1b77-4850-b6ad-960a92533271-kube-api-access-g8mxq\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.366000 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.365970 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e8d0ba1-1b77-4850-b6ad-960a92533271-tmp\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.366168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.366149 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8d0ba1-1b77-4850-b6ad-960a92533271-tls-certs\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.419779 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.419747 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mxq\" (UniqueName: \"kubernetes.io/projected/4e8d0ba1-1b77-4850-b6ad-960a92533271-kube-api-access-g8mxq\") pod \"kube-auth-proxy-6d64f9d85b-crltx\" (UID: \"4e8d0ba1-1b77-4850-b6ad-960a92533271\") " pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.468424 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.468388 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:42.539115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.539054 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" Nov 02 21:39:42.693685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.693638 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:42.911221 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.911148 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx"] Nov 02 21:39:42.913870 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:39:42.913844 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8d0ba1_1b77_4850_b6ad_960a92533271.slice/crio-d94e0a384713bd4fee80dae1553fbf84460df16ad68909b80d721fcb0cfb7722 WatchSource:0}: Error finding container d94e0a384713bd4fee80dae1553fbf84460df16ad68909b80d721fcb0cfb7722: Status 404 returned error can't find the container with id d94e0a384713bd4fee80dae1553fbf84460df16ad68909b80d721fcb0cfb7722 Nov 02 21:39:42.915772 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.915755 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:39:42.947233 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:42.947205 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:43.191941 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:43.191864 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:43.352933 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:43.352896 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:43.442021 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:43.441947 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" event={"ID":"4e8d0ba1-1b77-4850-b6ad-960a92533271","Type":"ContainerStarted","Data":"d94e0a384713bd4fee80dae1553fbf84460df16ad68909b80d721fcb0cfb7722"} Nov 02 21:39:44.360621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:44.360578 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:44.518233 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:44.518196 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:44.918941 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:44.918886 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk"] Nov 02 21:39:44.986297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:44.986262 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk"] Nov 02 21:39:44.986452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:44.986388 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.039315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.039284 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lcwpm\"" Nov 02 21:39:45.039491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.039474 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 21:39:45.039584 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.039572 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 21:39:45.088758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.088729 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sq5jk\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.088893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.088774 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6dq\" (UniqueName: \"kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq\") pod \"kuadrant-operator-controller-manager-678d87766f-sq5jk\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.189503 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.189421 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sq5jk\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.189503 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.189470 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6dq\" (UniqueName: \"kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq\") pod \"kuadrant-operator-controller-manager-678d87766f-sq5jk\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.189839 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.189819 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sq5jk\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.249898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.249866 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6dq\" (UniqueName: \"kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq\") pod \"kuadrant-operator-controller-manager-678d87766f-sq5jk\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.324683 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.324646 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:45.832365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:45.832340 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk"] Nov 02 21:39:45.834839 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:39:45.834811 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5549b0_1277_4bbe_9600_2a9eb00575a0.slice/crio-445c209fa8b456fdd46d7766aa668b4cff8ed2ceba0a2d6feacd31f263e48dd7 WatchSource:0}: Error finding container 445c209fa8b456fdd46d7766aa668b4cff8ed2ceba0a2d6feacd31f263e48dd7: Status 404 returned error can't find the container with id 445c209fa8b456fdd46d7766aa668b4cff8ed2ceba0a2d6feacd31f263e48dd7 Nov 02 21:39:46.003750 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:39:46.003720 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:39:46.455287 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:46.455252 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" event={"ID":"4e8d0ba1-1b77-4850-b6ad-960a92533271","Type":"ContainerStarted","Data":"5fd31f2ad2143ddc81a739d31915f884d0fbd406ce5ed11ec217d316bfef05a2"} Nov 02 21:39:46.456541 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:46.456519 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" event={"ID":"1b5549b0-1277-4bbe-9600-2a9eb00575a0","Type":"ContainerStarted","Data":"445c209fa8b456fdd46d7766aa668b4cff8ed2ceba0a2d6feacd31f263e48dd7"} Nov 02 21:39:46.567671 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:46.567621 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d64f9d85b-crltx" podStartSLOduration=1.651329262 podStartE2EDuration="4.56760621s" totalImagesPullingTime="2.916276948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:46.566850647 +0000 UTC m=+325.528400966" watchObservedRunningTime="2026-11-02 21:39:46.56760621 +0000 UTC m=+325.529156528" Nov 02 21:39:46.705646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:46.705579 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:46.961845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:46.961773 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:49.978918 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:49.978885 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:51.105035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:51.104996 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:51.299822 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:51.299794 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:51.453275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:51.453239 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:54.492285 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:54.492247 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" event={"ID":"1b5549b0-1277-4bbe-9600-2a9eb00575a0","Type":"ContainerStarted","Data":"7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4"} Nov 02 21:39:54.492695 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:54.492346 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:39:54.708787 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:54.708732 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" podStartSLOduration=2.697498242 podStartE2EDuration="10.708718581s" totalImagesPullingTime="8.011220339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:54.657688338 +0000 UTC m=+333.619238692" watchObservedRunningTime="2026-11-02 21:39:54.708718581 +0000 UTC m=+333.670268899" Nov 02 21:39:54.911026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:54.910949 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:55.057628 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:55.057597 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:55.593491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:55.593459 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:55.750663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:55.750633 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:59.457610 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:59.457579 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:59.612453 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:39:59.612420 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:01.997254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:01.997223 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:02.146749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:02.146693 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:02.303781 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:02.303715 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:03.175945 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:03.175918 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:03.340829 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:03.340799 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.124445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:04.124417 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.383766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:04.383683 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.606061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:04.606032 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.766411 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:04.766383 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:05.391604 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:05.391575 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:05.498583 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:05.498559 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:40:06.503662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:06.503630 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:06.662102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:06.662049 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:08.401686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:08.401654 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:11.289549 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:11.289508 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:11.459911 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:11.459874 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:13.913271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:13.913242 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:14.066141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:14.066098 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:16.004457 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:40:16.004422 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:40:19.290116 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:19.290084 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:19.438927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:19.438898 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:19.695656 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:19.695630 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:19.854163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:19.854106 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:20.092850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:20.092778 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:20.265670 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:20.265641 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:26.327615 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:26.327586 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:26.478462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:26.478428 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.136704 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:31.136671 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.293232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:31.293203 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:33.358763 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:33.358728 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:33.553702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:33.553657 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:33.979287 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:33.979252 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl"] Nov 02 21:40:33.985483 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:33.985460 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.038938 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.038909 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl"] Nov 02 21:40:34.086947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.086909 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:40:34.087088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.087000 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 21:40:34.087088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.087000 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:40:34.087088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.087056 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xxnrh\"" Nov 02 21:40:34.092325 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.092305 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:34.164653 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.164627 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f756716-a3b4-44e3-a23c-53860c6385d9-data\") pod \"postgres-7cdbf6dc44-hcdfl\" (UID: \"1f756716-a3b4-44e3-a23c-53860c6385d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.164771 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.164740 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zw2\" (UniqueName: \"kubernetes.io/projected/1f756716-a3b4-44e3-a23c-53860c6385d9-kube-api-access-x4zw2\") pod \"postgres-7cdbf6dc44-hcdfl\" (UID: \"1f756716-a3b4-44e3-a23c-53860c6385d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.265200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.265121 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zw2\" (UniqueName: \"kubernetes.io/projected/1f756716-a3b4-44e3-a23c-53860c6385d9-kube-api-access-x4zw2\") pod \"postgres-7cdbf6dc44-hcdfl\" (UID: \"1f756716-a3b4-44e3-a23c-53860c6385d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.265200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.265188 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f756716-a3b4-44e3-a23c-53860c6385d9-data\") pod \"postgres-7cdbf6dc44-hcdfl\" (UID: \"1f756716-a3b4-44e3-a23c-53860c6385d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.265518 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.265503 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f756716-a3b4-44e3-a23c-53860c6385d9-data\") pod \"postgres-7cdbf6dc44-hcdfl\" (UID: \"1f756716-a3b4-44e3-a23c-53860c6385d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.320956 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.320930 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zw2\" (UniqueName: \"kubernetes.io/projected/1f756716-a3b4-44e3-a23c-53860c6385d9-kube-api-access-x4zw2\") pod \"postgres-7cdbf6dc44-hcdfl\" (UID: \"1f756716-a3b4-44e3-a23c-53860c6385d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.596021 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.595945 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:34.978293 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:34.978263 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl"] Nov 02 21:40:34.982189 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:40:34.982161 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f756716_a3b4_44e3_a23c_53860c6385d9.slice/crio-c9730007e3ea923c23f3524f8997e0c0f0e680cd929b3f7a09a2f59f67a320d4 WatchSource:0}: Error finding container c9730007e3ea923c23f3524f8997e0c0f0e680cd929b3f7a09a2f59f67a320d4: Status 404 returned error can't find the container with id c9730007e3ea923c23f3524f8997e0c0f0e680cd929b3f7a09a2f59f67a320d4 Nov 02 21:40:35.657734 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:35.657699 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" event={"ID":"1f756716-a3b4-44e3-a23c-53860c6385d9","Type":"ContainerStarted","Data":"c9730007e3ea923c23f3524f8997e0c0f0e680cd929b3f7a09a2f59f67a320d4"} Nov 02 21:40:42.687910 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:42.687875 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" event={"ID":"1f756716-a3b4-44e3-a23c-53860c6385d9","Type":"ContainerStarted","Data":"0a70f6373699c354aa78a2aa10bd6fc446717624dee9dae8de988f23be4344e4"} Nov 02 21:40:42.688316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:42.688015 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:42.800708 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:42.800655 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" podStartSLOduration=3.003149673 podStartE2EDuration="9.80064237s" totalImagesPullingTime="6.797492697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:42.799235684 +0000 UTC m=+381.760786018" watchObservedRunningTime="2026-11-02 21:40:42.80064237 +0000 UTC m=+381.762192688" Nov 02 21:40:46.003322 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:40:46.003287 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:40:48.719027 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:48.719001 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hcdfl" Nov 02 21:40:55.468587 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:55.468554 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.626449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:55.626414 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.773237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:55.773160 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.944604 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:40:55.944566 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:00.105914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:00.105879 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:00.279294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:00.279259 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:03.642465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:03.642380 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:03.795277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:03.795246 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:04.468659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:04.468622 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:06.936455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:06.936424 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh"] Nov 02 21:41:07.037796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.037758 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.042501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.042440 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh"] Nov 02 21:41:07.116741 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116713 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.116894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116748 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.116894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116771 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.116894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116791 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.116994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116893 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdks\" (UniqueName: \"kubernetes.io/projected/c6611b39-cd2a-4a7a-a199-d198f1a94049-kube-api-access-kcdks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.116994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116961 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6611b39-cd2a-4a7a-a199-d198f1a94049-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.116994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.116987 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.217998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.217967 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6611b39-cd2a-4a7a-a199-d198f1a94049-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.217998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218000 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.218277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218033 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.218277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218053 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.218277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218071 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.218277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218092 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.218277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218110 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdks\" (UniqueName: \"kubernetes.io/projected/c6611b39-cd2a-4a7a-a199-d198f1a94049-kube-api-access-kcdks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.218956 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.218924 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.220506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.220477 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6611b39-cd2a-4a7a-a199-d198f1a94049-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.220679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.220659 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.220679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.220669 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.220792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.220763 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.277823 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.277793 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdks\" (UniqueName: \"kubernetes.io/projected/c6611b39-cd2a-4a7a-a199-d198f1a94049-kube-api-access-kcdks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.277962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.277834 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6611b39-cd2a-4a7a-a199-d198f1a94049-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q96hh\" (UID: \"c6611b39-cd2a-4a7a-a199-d198f1a94049\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.366322 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.366288 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:07.788200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.788170 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh"] Nov 02 21:41:07.826874 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:41:07.826845 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6611b39_cd2a_4a7a_a199_d198f1a94049.slice/crio-3c95ced36ea3670a61c91d3559b4b05bac3b5ed8748fe3fac02432622c9fbb38 WatchSource:0}: Error finding container 3c95ced36ea3670a61c91d3559b4b05bac3b5ed8748fe3fac02432622c9fbb38: Status 404 returned error can't find the container with id 3c95ced36ea3670a61c91d3559b4b05bac3b5ed8748fe3fac02432622c9fbb38 Nov 02 21:41:07.828843 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.828814 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:41:07.828932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:07.828873 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:41:08.789292 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:08.789253 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" event={"ID":"c6611b39-cd2a-4a7a-a199-d198f1a94049","Type":"ContainerStarted","Data":"6fef2d983fb01d9a136cc174b3c776bd26c1499b565f396b2a612f080fecda62"} Nov 02 21:41:08.789756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:08.789305 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" event={"ID":"c6611b39-cd2a-4a7a-a199-d198f1a94049","Type":"ContainerStarted","Data":"3c95ced36ea3670a61c91d3559b4b05bac3b5ed8748fe3fac02432622c9fbb38"} Nov 02 21:41:08.789756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:08.789470 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:08.791291 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:08.791264 4285 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-q96hh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:41:08.791407 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:08.791319 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" podUID="c6611b39-cd2a-4a7a-a199-d198f1a94049" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:41:09.014568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:09.014502 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" podStartSLOduration=3.014485542 podStartE2EDuration="3.014485542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:08.963823575 +0000 UTC m=+407.925373896" watchObservedRunningTime="2026-11-02 21:41:09.014485542 +0000 UTC m=+407.976035861" Nov 02 21:41:09.796166 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:09.796119 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q96hh" Nov 02 21:41:10.015896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.015858 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd"] Nov 02 21:41:10.016095 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.016058 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerName="discovery" containerID="cri-o://f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea" gracePeriod=30 Nov 02 21:41:10.253470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.253435 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:41:10.340831 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.340796 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.340865 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.340888 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.340915 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.340934 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.340973 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.341014 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token\") pod \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\" (UID: \"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e\") " Nov 02 21:41:10.341466 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.341431 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:10.343443 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.343415 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:10.343550 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.343453 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.343646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.343621 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "kube-api-access-sv5vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:10.343714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.343641 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:10.343714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.343668 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:10.343814 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.343794 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs" pod "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" (UID: "3d307a43-fd7d-4b9a-aebe-7c00cb169b2e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:10.442669 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442639 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.442669 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442670 4285 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-kubeconfig\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.442868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442685 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv5vk\" (UniqueName: \"kubernetes.io/projected/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-kube-api-access-sv5vk\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.442868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442699 4285 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-local-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.442868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442711 4285 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-ca-configmap\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.442868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442721 4285 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-istio-csr-dns-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.442868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.442734 4285 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e-cacerts\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.800621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.800524 4285 generic.go:356] "Generic (PLEG): container finished" podID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerID="f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea" exitCode=0 Nov 02 21:41:10.800621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.800578 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" Nov 02 21:41:10.801115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.800616 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" event={"ID":"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e","Type":"ContainerDied","Data":"f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea"} Nov 02 21:41:10.801115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.800679 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd" event={"ID":"3d307a43-fd7d-4b9a-aebe-7c00cb169b2e","Type":"ContainerDied","Data":"316cd476e7fe2dc9f4ac92dd121a4958722405a3d1c5d8e50f45403e0c470629"} Nov 02 21:41:10.801115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.800711 4285 scope.go:122] "RemoveContainer" containerID="f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea" Nov 02 21:41:10.809173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.809157 4285 scope.go:122] "RemoveContainer" containerID="f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea" Nov 02 21:41:10.809440 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:41:10.809422 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea\": container with ID starting with f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea not found: ID does not exist" containerID="f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea" Nov 02 21:41:10.809506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.809451 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea"} err="failed to get container status \"f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea\": rpc error: code = NotFound desc = could not find container \"f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea\": container with ID starting with f8f66eb7d16c515ced7c8b2b9ca300911202290532a99aab7ea3a6f83aa36aea not found: ID does not exist" Nov 02 21:41:10.962884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.962846 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd"] Nov 02 21:41:10.966093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:10.966065 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-h99jd"] Nov 02 21:41:11.171437 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:11.171358 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" path="/var/lib/kubelet/pods/3d307a43-fd7d-4b9a-aebe-7c00cb169b2e/volumes" Nov 02 21:41:12.120216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:12.120177 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:12.279080 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:12.279048 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:12.436054 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:12.435969 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:12.587877 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:12.587844 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:12.740288 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:12.740260 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:12.892524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:12.892494 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:13.051169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.051081 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:13.178602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.178567 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pbvwm"] Nov 02 21:41:13.179044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.179028 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerName="discovery" Nov 02 21:41:13.179144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.179045 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerName="discovery" Nov 02 21:41:13.179202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.179143 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d307a43-fd7d-4b9a-aebe-7c00cb169b2e" containerName="discovery" Nov 02 21:41:13.181548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.181526 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.211888 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.211860 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:13.266336 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.266301 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/cc7a35e2-d3a3-4f3a-a278-84423b5ea520-kube-api-access-9hk6l\") pod \"model-serving-api-75bd89fdfc-pbvwm\" (UID: \"cc7a35e2-d3a3-4f3a-a278-84423b5ea520\") " pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.266494 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.266393 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7a35e2-d3a3-4f3a-a278-84423b5ea520-tls-certs\") pod \"model-serving-api-75bd89fdfc-pbvwm\" (UID: \"cc7a35e2-d3a3-4f3a-a278-84423b5ea520\") " pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.283372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.283341 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-72fsd\"" Nov 02 21:41:13.283474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.283386 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 21:41:13.283474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.283435 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 21:41:13.283474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.283443 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 21:41:13.284757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.284739 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pbvwm"] Nov 02 21:41:13.367375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.367304 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/cc7a35e2-d3a3-4f3a-a278-84423b5ea520-kube-api-access-9hk6l\") pod \"model-serving-api-75bd89fdfc-pbvwm\" (UID: \"cc7a35e2-d3a3-4f3a-a278-84423b5ea520\") " pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.367375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.367340 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7a35e2-d3a3-4f3a-a278-84423b5ea520-tls-certs\") pod \"model-serving-api-75bd89fdfc-pbvwm\" (UID: \"cc7a35e2-d3a3-4f3a-a278-84423b5ea520\") " pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.369773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.369751 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc7a35e2-d3a3-4f3a-a278-84423b5ea520-tls-certs\") pod \"model-serving-api-75bd89fdfc-pbvwm\" (UID: \"cc7a35e2-d3a3-4f3a-a278-84423b5ea520\") " pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.424092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.424061 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/cc7a35e2-d3a3-4f3a-a278-84423b5ea520-kube-api-access-9hk6l\") pod \"model-serving-api-75bd89fdfc-pbvwm\" (UID: \"cc7a35e2-d3a3-4f3a-a278-84423b5ea520\") " pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.492211 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.492182 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:13.857373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:13.857349 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pbvwm"] Nov 02 21:41:13.860210 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:41:13.860181 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7a35e2_d3a3_4f3a_a278_84423b5ea520.slice/crio-ed69865e95cc2c48a50509349615fffaf8a7d72cb576dd470a5eacaea713f4ea WatchSource:0}: Error finding container ed69865e95cc2c48a50509349615fffaf8a7d72cb576dd470a5eacaea713f4ea: Status 404 returned error can't find the container with id ed69865e95cc2c48a50509349615fffaf8a7d72cb576dd470a5eacaea713f4ea Nov 02 21:41:14.316463 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:14.316433 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:14.819086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:14.819051 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" event={"ID":"cc7a35e2-d3a3-4f3a-a278-84423b5ea520","Type":"ContainerStarted","Data":"ed69865e95cc2c48a50509349615fffaf8a7d72cb576dd470a5eacaea713f4ea"} Nov 02 21:41:16.005411 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:41:16.005376 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:41:16.829711 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:16.829675 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" event={"ID":"cc7a35e2-d3a3-4f3a-a278-84423b5ea520","Type":"ContainerStarted","Data":"a31ccce33eb43ca46a3a209109e90dbc419f0e8ce7342dd3d87e403e3c699d01"} Nov 02 21:41:16.829899 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:16.829796 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:16.943434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:16.943377 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" podStartSLOduration=1.3672480839999999 podStartE2EDuration="3.943359728s" totalImagesPullingTime="2.576111644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:16.941506332 +0000 UTC m=+415.903056651" watchObservedRunningTime="2026-11-02 21:41:16.943359728 +0000 UTC m=+415.904910083" Nov 02 21:41:19.254527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:19.254493 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:19.412024 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:19.411985 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:19.566008 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:19.565931 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:19.704843 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:19.704814 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:19.854896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:19.854819 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:20.002892 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:20.002855 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:20.148892 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:20.148803 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:20.294142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:20.294100 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:20.445389 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:20.445350 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:27.837924 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:27.837896 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-pbvwm" Nov 02 21:41:33.494299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:33.494265 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:33.657862 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:33.657823 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:35.455711 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:35.455676 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:35.654096 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:35.654045 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:35.802431 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:35.802320 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:35.952782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:35.952748 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:36.542482 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.542446 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp"] Nov 02 21:41:36.545639 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.545615 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.571889 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.571866 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.571995 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.571918 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.572045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.572025 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.572081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.572053 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xgp\" (UniqueName: \"kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.647390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.647351 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:41:36.647390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.647387 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 21:41:36.647587 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.647490 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cbb6p\"" Nov 02 21:41:36.648944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.648926 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp"] Nov 02 21:41:36.672905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.672880 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.673005 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.672915 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xgp\" (UniqueName: \"kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.673005 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.672970 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.673068 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.673014 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.673687 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.673667 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.675492 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.675471 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.675574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.675504 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.729013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.728979 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xgp\" (UniqueName: \"kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp\") pod \"maas-api-6d669c7fcd-tdcjp\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:36.857205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:36.857104 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:37.223048 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:37.223024 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp"] Nov 02 21:41:37.225445 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:41:37.225418 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dc2a90_7aa3_4292_bea7_84be252265f9.slice/crio-1bf61d970fa5ca5ebb2828cbfe99694f1566eee9bfc28c5082166db7d7a8b07e WatchSource:0}: Error finding container 1bf61d970fa5ca5ebb2828cbfe99694f1566eee9bfc28c5082166db7d7a8b07e: Status 404 returned error can't find the container with id 1bf61d970fa5ca5ebb2828cbfe99694f1566eee9bfc28c5082166db7d7a8b07e Nov 02 21:41:37.915752 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:37.915716 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" event={"ID":"a9dc2a90-7aa3-4292-bea7-84be252265f9","Type":"ContainerStarted","Data":"1bf61d970fa5ca5ebb2828cbfe99694f1566eee9bfc28c5082166db7d7a8b07e"} Nov 02 21:41:40.930924 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:40.930889 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" event={"ID":"a9dc2a90-7aa3-4292-bea7-84be252265f9","Type":"ContainerStarted","Data":"6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588"} Nov 02 21:41:40.931338 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:40.930967 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:41.044648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:41.044589 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" podStartSLOduration=2.184558115 podStartE2EDuration="5.044573418s" totalImagesPullingTime="2.860015303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:41.043177137 +0000 UTC m=+440.004727458" watchObservedRunningTime="2026-11-02 21:41:41.044573418 +0000 UTC m=+440.006123738" Nov 02 21:41:46.004732 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:41:46.004686 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:41:46.940562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:46.940535 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:47.469987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:47.469957 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:47.859724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:47.859646 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:57.439783 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.439747 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp"] Nov 02 21:41:57.440221 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.439927 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" podUID="a9dc2a90-7aa3-4292-bea7-84be252265f9" containerName="maas-api" containerID="cri-o://6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588" gracePeriod=30 Nov 02 21:41:57.673973 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.673950 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:57.752050 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.752021 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca\") pod \"a9dc2a90-7aa3-4292-bea7-84be252265f9\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " Nov 02 21:41:57.752219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.752068 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs\") pod \"a9dc2a90-7aa3-4292-bea7-84be252265f9\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " Nov 02 21:41:57.752219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.752166 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls\") pod \"a9dc2a90-7aa3-4292-bea7-84be252265f9\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " Nov 02 21:41:57.752219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.752200 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp\" (UniqueName: \"kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp\") pod \"a9dc2a90-7aa3-4292-bea7-84be252265f9\" (UID: \"a9dc2a90-7aa3-4292-bea7-84be252265f9\") " Nov 02 21:41:57.752394 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.752371 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca" pod "a9dc2a90-7aa3-4292-bea7-84be252265f9" (UID: "a9dc2a90-7aa3-4292-bea7-84be252265f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:57.752488 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.752473 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9dc2a90-7aa3-4292-bea7-84be252265f9-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:57.754266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.754240 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs" pod "a9dc2a90-7aa3-4292-bea7-84be252265f9" (UID: "a9dc2a90-7aa3-4292-bea7-84be252265f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:57.754356 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.754287 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls" pod "a9dc2a90-7aa3-4292-bea7-84be252265f9" (UID: "a9dc2a90-7aa3-4292-bea7-84be252265f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:57.754356 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.754301 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp" pod "a9dc2a90-7aa3-4292-bea7-84be252265f9" (UID: "a9dc2a90-7aa3-4292-bea7-84be252265f9"). InnerVolumeSpecName "kube-api-access-z7xgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:57.853676 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.853646 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:57.853676 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.853669 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xgp\" (UniqueName: \"kubernetes.io/projected/a9dc2a90-7aa3-4292-bea7-84be252265f9-kube-api-access-z7xgp\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:57.853676 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:57.853680 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9dc2a90-7aa3-4292-bea7-84be252265f9-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:41:58.003359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.003279 4285 generic.go:356] "Generic (PLEG): container finished" podID="a9dc2a90-7aa3-4292-bea7-84be252265f9" containerID="6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588" exitCode=0 Nov 02 21:41:58.003359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.003327 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" Nov 02 21:41:58.003359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.003333 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" event={"ID":"a9dc2a90-7aa3-4292-bea7-84be252265f9","Type":"ContainerDied","Data":"6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588"} Nov 02 21:41:58.003561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.003385 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp" event={"ID":"a9dc2a90-7aa3-4292-bea7-84be252265f9","Type":"ContainerDied","Data":"1bf61d970fa5ca5ebb2828cbfe99694f1566eee9bfc28c5082166db7d7a8b07e"} Nov 02 21:41:58.003561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.003414 4285 scope.go:122] "RemoveContainer" containerID="6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588" Nov 02 21:41:58.012596 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.012575 4285 scope.go:122] "RemoveContainer" containerID="6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588" Nov 02 21:41:58.012850 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:41:58.012827 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588\": container with ID starting with 6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588 not found: ID does not exist" containerID="6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588" Nov 02 21:41:58.012910 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.012858 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588"} err="failed to get container status \"6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588\": rpc error: code = NotFound desc = could not find container \"6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588\": container with ID starting with 6f613fa83e431c7172a883f8ef58f0fc9c72b4973a492102159f4d9da610a588 not found: ID does not exist" Nov 02 21:41:58.163641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.163615 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp"] Nov 02 21:41:58.213229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:58.213203 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tdcjp"] Nov 02 21:41:59.172328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:41:59.172292 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9dc2a90-7aa3-4292-bea7-84be252265f9" path="/var/lib/kubelet/pods/a9dc2a90-7aa3-4292-bea7-84be252265f9/volumes" Nov 02 21:42:04.808243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:42:04.808200 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:16.004858 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:42:16.004822 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:42:18.402732 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:42:18.402704 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:38.797716 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:42:38.797628 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:46.003522 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:42:46.003482 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:43:05.138562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:05.138527 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:05.390596 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:05.390509 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:05.869190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:05.869161 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:06.016494 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:06.016464 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:06.454249 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:06.454210 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:06.612576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:06.612538 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:12.079996 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:12.079946 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:16.003903 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:43:16.003867 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:43:24.062359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.062322 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x"] Nov 02 21:43:24.062792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.062616 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a9dc2a90-7aa3-4292-bea7-84be252265f9" containerName="maas-api" Nov 02 21:43:24.062792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.062626 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9dc2a90-7aa3-4292-bea7-84be252265f9" containerName="maas-api" Nov 02 21:43:24.062792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.062697 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9dc2a90-7aa3-4292-bea7-84be252265f9" containerName="maas-api" Nov 02 21:43:24.065527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.065511 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.167542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.167504 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 21:43:24.167756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.167598 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jr9p4\"" Nov 02 21:43:24.167756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.167622 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 21:43:24.167756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.167631 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 21:43:24.169300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.169282 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x"] Nov 02 21:43:24.198413 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.198391 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:24.218539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.218508 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.218685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.218549 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsktq\" (UniqueName: \"kubernetes.io/projected/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-kube-api-access-bsktq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.218685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.218588 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.218900 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.218878 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.218992 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.218977 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.219040 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.219007 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.319722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.319630 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.319722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.319676 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.319722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.319714 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.319722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.319730 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsktq\" (UniqueName: \"kubernetes.io/projected/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-kube-api-access-bsktq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.320045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.319752 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.320045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.319786 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.320175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.320050 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.320175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.320103 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.320175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.320164 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.322035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.322009 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.322143 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.322112 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.376709 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.376675 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsktq\" (UniqueName: \"kubernetes.io/projected/6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41-kube-api-access-bsktq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x\" (UID: \"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.376854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.376839 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:24.752540 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:24.752516 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x"] Nov 02 21:43:24.755039 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:43:24.755015 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8ab551_6d4a_4ad1_a3c5_c662ed0adc41.slice/crio-ddde1f31500e9b4ac47f922b121a645b37c0145c5445abb0ba2c4851b5cd7655 WatchSource:0}: Error finding container ddde1f31500e9b4ac47f922b121a645b37c0145c5445abb0ba2c4851b5cd7655: Status 404 returned error can't find the container with id ddde1f31500e9b4ac47f922b121a645b37c0145c5445abb0ba2c4851b5cd7655 Nov 02 21:43:25.348000 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:25.347961 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" event={"ID":"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41","Type":"ContainerStarted","Data":"ddde1f31500e9b4ac47f922b121a645b37c0145c5445abb0ba2c4851b5cd7655"} Nov 02 21:43:27.358259 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:27.358171 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" event={"ID":"6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41","Type":"ContainerStarted","Data":"df4b86e4fe9721110abc570ac267519efd867c512f2734466a92c0fe4d5e9ec5"} Nov 02 21:43:27.376974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:27.376946 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:27.472259 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:27.472206 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" podStartSLOduration=1.281580185 podStartE2EDuration="3.472189544s" totalImagesPullingTime="2.190609359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:27.471514787 +0000 UTC m=+546.433065107" watchObservedRunningTime="2026-11-02 21:43:27.472189544 +0000 UTC m=+546.433739866" Nov 02 21:43:29.694658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:29.694631 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:34.076138 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.076090 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6"] Nov 02 21:43:34.079657 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.079636 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.180704 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.180673 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 21:43:34.182289 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.182269 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6"] Nov 02 21:43:34.198322 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.198295 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:34.211419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.211399 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.211485 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.211425 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.211485 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.211455 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtntq\" (UniqueName: \"kubernetes.io/projected/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-kube-api-access-rtntq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.211600 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.211583 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.211635 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.211621 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.211670 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.211637 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.312721 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.312686 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.312721 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.312723 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.312939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.312771 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtntq\" (UniqueName: \"kubernetes.io/projected/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-kube-api-access-rtntq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.312939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.312813 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.312939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.312867 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.312939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.312894 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.313221 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.313195 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.313353 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.313286 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.313452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.313359 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.314875 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.314857 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.315161 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.315145 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.368580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.368521 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtntq\" (UniqueName: \"kubernetes.io/projected/cc6364ff-f21b-462b-a93e-f71e85c3e0f8-kube-api-access-rtntq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6\" (UID: \"cc6364ff-f21b-462b-a93e-f71e85c3e0f8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.377354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.377333 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:34.389441 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.389416 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:34.390976 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.390957 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:34.401731 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.401715 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x" Nov 02 21:43:34.757780 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:34.757755 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6"] Nov 02 21:43:34.759731 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:43:34.759711 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6364ff_f21b_462b_a93e_f71e85c3e0f8.slice/crio-2919213dcbc8f2ede0fe377e39ab1bf8aaa5722516c159863fb423fb5467fabe WatchSource:0}: Error finding container 2919213dcbc8f2ede0fe377e39ab1bf8aaa5722516c159863fb423fb5467fabe: Status 404 returned error can't find the container with id 2919213dcbc8f2ede0fe377e39ab1bf8aaa5722516c159863fb423fb5467fabe Nov 02 21:43:35.392162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:35.392135 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" event={"ID":"cc6364ff-f21b-462b-a93e-f71e85c3e0f8","Type":"ContainerStarted","Data":"2919213dcbc8f2ede0fe377e39ab1bf8aaa5722516c159863fb423fb5467fabe"} Nov 02 21:43:36.398360 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:36.398324 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" event={"ID":"cc6364ff-f21b-462b-a93e-f71e85c3e0f8","Type":"ContainerStarted","Data":"1355df2673717ffd6e82ac769f230e8c5630edd30e6be3516abd2e5fd2e342c0"} Nov 02 21:43:36.513797 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:36.513749 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" podStartSLOduration=1.9232181160000001 podStartE2EDuration="2.513734998s" totalImagesPullingTime="590.516882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:36.512611659 +0000 UTC m=+555.474161992" watchObservedRunningTime="2026-11-02 21:43:36.513734998 +0000 UTC m=+555.475285315" Nov 02 21:43:37.391367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:37.391337 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:37.640526 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:37.640494 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:41.246093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.245970 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887"] Nov 02 21:43:41.259574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.249256 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.260298 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.260275 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.260415 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.260305 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-kube-api-access-dl8rn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.260415 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.260326 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.260487 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.260468 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.260533 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.260520 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.260566 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.260545 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.350341 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.350308 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 21:43:41.352452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.352420 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887"] Nov 02 21:43:41.360973 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.360940 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361206 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-kube-api-access-dl8rn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361263 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361320 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361477 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361360 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361477 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361367 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361477 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361418 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361709 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361691 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.361986 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.361929 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.364100 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.364077 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.364491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.364469 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.368656 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.368636 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:41.416982 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.416960 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/1f8b5643-8dd6-45ee-ba29-6d6d573eaad6-kube-api-access-dl8rn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tw887\" (UID: \"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.561051 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.560966 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:41.925195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:41.925173 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887"] Nov 02 21:43:41.927371 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:43:41.927345 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8b5643_8dd6_45ee_ba29_6d6d573eaad6.slice/crio-f661df7823b26cbe260b2c0391c2810ae639288647a43670191e70f0779aad5e WatchSource:0}: Error finding container f661df7823b26cbe260b2c0391c2810ae639288647a43670191e70f0779aad5e: Status 404 returned error can't find the container with id f661df7823b26cbe260b2c0391c2810ae639288647a43670191e70f0779aad5e Nov 02 21:43:42.427371 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:42.427338 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" event={"ID":"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6","Type":"ContainerStarted","Data":"f661df7823b26cbe260b2c0391c2810ae639288647a43670191e70f0779aad5e"} Nov 02 21:43:43.432856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:43.432821 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" event={"ID":"1f8b5643-8dd6-45ee-ba29-6d6d573eaad6","Type":"ContainerStarted","Data":"a64b543a0f5b8c6821d973313876f503ec60aa477719af26585d811c9732b01f"} Nov 02 21:43:43.547443 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:43.547391 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" podStartSLOduration=1.9769939079999999 podStartE2EDuration="2.54737699s" totalImagesPullingTime="570.383082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:43.54499993 +0000 UTC m=+562.506550250" watchObservedRunningTime="2026-11-02 21:43:43.54737699 +0000 UTC m=+562.508927353" Nov 02 21:43:43.561202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:43.561176 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:44.391215 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:44.391180 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:44.403231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:44.403203 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:44.447845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:44.447820 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" Nov 02 21:43:45.395560 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:45.395530 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:46.004112 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:43:46.004077 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:43:49.436324 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.436288 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp"] Nov 02 21:43:49.439940 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.439915 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.532899 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.532859 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.533056 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.532984 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.533056 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.533016 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcz5\" (UniqueName: \"kubernetes.io/projected/615e6adb-9482-405a-85f3-98d3af5df4cd-kube-api-access-6gcz5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.533056 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.533045 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615e6adb-9482-405a-85f3-98d3af5df4cd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.533212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.533112 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.533212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.533196 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.544710 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.544681 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 21:43:49.546237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.546211 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp"] Nov 02 21:43:49.546757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.546739 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:49.634354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634314 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634371 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcz5\" (UniqueName: \"kubernetes.io/projected/615e6adb-9482-405a-85f3-98d3af5df4cd-kube-api-access-6gcz5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634420 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615e6adb-9482-405a-85f3-98d3af5df4cd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634457 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634497 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634520 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634955 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634917 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.634955 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634953 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.635075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.634978 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.636780 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.636751 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615e6adb-9482-405a-85f3-98d3af5df4cd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.636898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.636882 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615e6adb-9482-405a-85f3-98d3af5df4cd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.694543 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.694507 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcz5\" (UniqueName: \"kubernetes.io/projected/615e6adb-9482-405a-85f3-98d3af5df4cd-kube-api-access-6gcz5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp\" (UID: \"615e6adb-9482-405a-85f3-98d3af5df4cd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:49.751915 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:49.751886 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:50.116777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:50.116747 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp"] Nov 02 21:43:50.119071 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:43:50.119044 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615e6adb_9482_405a_85f3_98d3af5df4cd.slice/crio-56507cd73f09904807aa79255faf59af253c9f03a646715f2f82b02b46b0c6d4 WatchSource:0}: Error finding container 56507cd73f09904807aa79255faf59af253c9f03a646715f2f82b02b46b0c6d4: Status 404 returned error can't find the container with id 56507cd73f09904807aa79255faf59af253c9f03a646715f2f82b02b46b0c6d4 Nov 02 21:43:50.463028 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:50.462989 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" event={"ID":"615e6adb-9482-405a-85f3-98d3af5df4cd","Type":"ContainerStarted","Data":"56507cd73f09904807aa79255faf59af253c9f03a646715f2f82b02b46b0c6d4"} Nov 02 21:43:51.468790 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:51.468753 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" event={"ID":"615e6adb-9482-405a-85f3-98d3af5df4cd","Type":"ContainerStarted","Data":"96e38a2c9fb8be07b08cebdaf2537dbc73667235b3b24c57e34fa95bfc9cc4d5"} Nov 02 21:43:51.561533 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:51.561496 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:51.574354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:51.574324 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:51.583475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:51.583426 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" podStartSLOduration=1.9304027289999999 podStartE2EDuration="2.583411111s" totalImagesPullingTime="653.008382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:51.581415115 +0000 UTC m=+570.542965433" watchObservedRunningTime="2026-11-02 21:43:51.583411111 +0000 UTC m=+570.544961430" Nov 02 21:43:51.752270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:51.752188 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:52.483750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:52.483720 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" Nov 02 21:43:52.998367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:52.998336 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:55.681245 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:55.681213 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:55.829763 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:55.829732 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:59.752137 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:59.752106 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:43:59.764341 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:43:59.764314 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:44:00.516186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:00.516158 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" Nov 02 21:44:03.621276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.621191 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448"] Nov 02 21:44:03.627184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.627163 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.679414 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.679381 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qzh4z\"" Nov 02 21:44:03.762436 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762402 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762436 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762444 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762531 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fwp\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762574 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762597 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762616 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762733 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762764 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.762884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.762781 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.830565 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.830536 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448"] Nov 02 21:44:03.863268 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863240 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863268 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863274 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863331 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863364 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863427 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fwp\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863453 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863712 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863488 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863712 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863529 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863712 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863649 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863712 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863662 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.863954 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.863932 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.864324 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.864298 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.864427 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.864331 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.864427 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.864412 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.866344 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.866320 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.866449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.866431 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.919457 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.919396 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.919550 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.919470 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fwp\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp\") pod \"maas-default-gateway-openshift-default-65c448b547-nw448\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:03.939372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:03.939348 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:04.412644 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:04.412621 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448"] Nov 02 21:44:04.414540 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:44:04.414516 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2722e3ac_1614_4185_a842_d6b27f4c86e1.slice/crio-89dcd6ee600e670226cb8559637cc0868be1faa54aaf6ff7bf9bd01260ee0558 WatchSource:0}: Error finding container 89dcd6ee600e670226cb8559637cc0868be1faa54aaf6ff7bf9bd01260ee0558: Status 404 returned error can't find the container with id 89dcd6ee600e670226cb8559637cc0868be1faa54aaf6ff7bf9bd01260ee0558 Nov 02 21:44:04.524649 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:04.524619 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" event={"ID":"2722e3ac-1614-4185-a842-d6b27f4c86e1","Type":"ContainerStarted","Data":"89dcd6ee600e670226cb8559637cc0868be1faa54aaf6ff7bf9bd01260ee0558"} Nov 02 21:44:05.479348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:05.479309 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:08.126486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.126441 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:08.239267 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.239233 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:08.239374 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.239301 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:08.239374 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.239333 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:08.543791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.543755 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" event={"ID":"2722e3ac-1614-4185-a842-d6b27f4c86e1","Type":"ContainerStarted","Data":"28c01a98052af08a2bf7d994021ef4e2ed06853dff906dee345630c4c54dff99"} Nov 02 21:44:08.709163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.708079 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podStartSLOduration=1.886152968 podStartE2EDuration="5.70806059s" totalImagesPullingTime="3.821907622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:08.707038621 +0000 UTC m=+587.668588954" watchObservedRunningTime="2026-11-02 21:44:08.70806059 +0000 UTC m=+587.669610911" Nov 02 21:44:08.940082 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:08.939975 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:09.941585 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:09.941544 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:44:09.942038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:09.941608 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:44:10.940422 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:10.940386 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:44:10.940601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:10.940449 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:44:11.940017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:11.939984 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:44:11.940453 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:11.940038 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:44:12.940795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:12.940756 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:44:12.941236 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:12.940826 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:44:13.940817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:13.940772 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:44:13.941234 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:13.940842 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:44:14.940605 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:14.940562 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:44:14.940818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:14.940629 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:44:16.007635 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:44:16.007595 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:44:16.434018 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:16.433934 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:16.434621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:16.434600 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:16.435600 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:16.435578 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:44:21.192590 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.192564 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-qc7ww"] Nov 02 21:44:21.196588 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.196564 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.204927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.204902 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:21.239061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.239028 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-qc7ww\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.239257 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.239221 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f58\" (UniqueName: \"kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58\") pod \"ai-gateway-controller-c7987c6b6-qc7ww\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.255922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.255886 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-qc7ww"] Nov 02 21:44:21.304460 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.304425 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 21:44:21.304460 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.304451 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 21:44:21.304460 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.304431 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-grzw5\"" Nov 02 21:44:21.339698 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.339668 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f58\" (UniqueName: \"kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58\") pod \"ai-gateway-controller-c7987c6b6-qc7ww\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.339863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.339719 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-qc7ww\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.342094 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.342065 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs\") pod \"ai-gateway-controller-c7987c6b6-qc7ww\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.397249 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.397217 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f58\" (UniqueName: \"kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58\") pod \"ai-gateway-controller-c7987c6b6-qc7ww\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.442568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.442538 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:21.517192 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.517073 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:21.885578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:21.885551 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-qc7ww"] Nov 02 21:44:21.887920 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:44:21.887888 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320cdede_8be0_4443_bd6a_640757afe583.slice/crio-456f0e53e1904502d7356b3ebd90e55648d8172a5717f43b9283ddb2183268b4 WatchSource:0}: Error finding container 456f0e53e1904502d7356b3ebd90e55648d8172a5717f43b9283ddb2183268b4: Status 404 returned error can't find the container with id 456f0e53e1904502d7356b3ebd90e55648d8172a5717f43b9283ddb2183268b4 Nov 02 21:44:22.597627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:22.597599 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:22.608272 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:22.606562 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" event={"ID":"320cdede-8be0-4443-bd6a-640757afe583","Type":"ContainerStarted","Data":"456f0e53e1904502d7356b3ebd90e55648d8172a5717f43b9283ddb2183268b4"} Nov 02 21:44:22.746696 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:22.746665 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:27.632254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:27.632206 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" event={"ID":"320cdede-8be0-4443-bd6a-640757afe583","Type":"ContainerStarted","Data":"f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46"} Nov 02 21:44:27.632631 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:27.632308 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:27.745435 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:27.745379 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" podStartSLOduration=1.7014416570000002 podStartE2EDuration="6.745363907s" totalImagesPullingTime="5.04392225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:27.743555636 +0000 UTC m=+606.705105957" watchObservedRunningTime="2026-11-02 21:44:27.745363907 +0000 UTC m=+606.706914226" Nov 02 21:44:30.879333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:30.879293 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:31.039231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.039171 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:31.215693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.215662 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f"] Nov 02 21:44:31.219058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.219041 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.278447 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.278416 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f"] Nov 02 21:44:31.320001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.319958 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 21:44:31.320186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.320088 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-knq2c\"" Nov 02 21:44:31.320237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.320223 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34cdf2d8-2345-46fe-b006-fb6f05a86bf2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rdz8f\" (UID: \"34cdf2d8-2345-46fe-b006-fb6f05a86bf2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.320271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.320246 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndwt\" (UniqueName: \"kubernetes.io/projected/34cdf2d8-2345-46fe-b006-fb6f05a86bf2-kube-api-access-mndwt\") pod \"payload-pre-processing-7b98bb678c-rdz8f\" (UID: \"34cdf2d8-2345-46fe-b006-fb6f05a86bf2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.372059 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.372028 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9l4v9"] Nov 02 21:44:31.375845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.375821 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.421505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.421458 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6730dff9-7d15-4a6a-bb30-63bccf531f4c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9l4v9\" (UID: \"6730dff9-7d15-4a6a-bb30-63bccf531f4c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.421714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.421535 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34cdf2d8-2345-46fe-b006-fb6f05a86bf2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rdz8f\" (UID: \"34cdf2d8-2345-46fe-b006-fb6f05a86bf2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.421714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.421579 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndwt\" (UniqueName: \"kubernetes.io/projected/34cdf2d8-2345-46fe-b006-fb6f05a86bf2-kube-api-access-mndwt\") pod \"payload-pre-processing-7b98bb678c-rdz8f\" (UID: \"34cdf2d8-2345-46fe-b006-fb6f05a86bf2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.421714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.421634 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnrl\" (UniqueName: \"kubernetes.io/projected/6730dff9-7d15-4a6a-bb30-63bccf531f4c-kube-api-access-mmnrl\") pod \"payload-processing-55fbbbf9dc-9l4v9\" (UID: \"6730dff9-7d15-4a6a-bb30-63bccf531f4c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.422309 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.422291 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34cdf2d8-2345-46fe-b006-fb6f05a86bf2-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rdz8f\" (UID: \"34cdf2d8-2345-46fe-b006-fb6f05a86bf2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.427101 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.427009 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9l4v9"] Nov 02 21:44:31.478142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.478057 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndwt\" (UniqueName: \"kubernetes.io/projected/34cdf2d8-2345-46fe-b006-fb6f05a86bf2-kube-api-access-mndwt\") pod \"payload-pre-processing-7b98bb678c-rdz8f\" (UID: \"34cdf2d8-2345-46fe-b006-fb6f05a86bf2\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.522473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.522446 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6730dff9-7d15-4a6a-bb30-63bccf531f4c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9l4v9\" (UID: \"6730dff9-7d15-4a6a-bb30-63bccf531f4c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.522626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.522506 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnrl\" (UniqueName: \"kubernetes.io/projected/6730dff9-7d15-4a6a-bb30-63bccf531f4c-kube-api-access-mmnrl\") pod \"payload-processing-55fbbbf9dc-9l4v9\" (UID: \"6730dff9-7d15-4a6a-bb30-63bccf531f4c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.523181 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.523161 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6730dff9-7d15-4a6a-bb30-63bccf531f4c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9l4v9\" (UID: \"6730dff9-7d15-4a6a-bb30-63bccf531f4c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.531297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.531276 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:31.578971 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.578733 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnrl\" (UniqueName: \"kubernetes.io/projected/6730dff9-7d15-4a6a-bb30-63bccf531f4c-kube-api-access-mmnrl\") pod \"payload-processing-55fbbbf9dc-9l4v9\" (UID: \"6730dff9-7d15-4a6a-bb30-63bccf531f4c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.694742 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.694705 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:31.899022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:31.898924 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f"] Nov 02 21:44:31.902580 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:44:31.902551 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cdf2d8_2345_46fe_b006_fb6f05a86bf2.slice/crio-123d2c924b9f85968d604347259f9e14263dc0754b98422afb8f0b2055aafdeb WatchSource:0}: Error finding container 123d2c924b9f85968d604347259f9e14263dc0754b98422afb8f0b2055aafdeb: Status 404 returned error can't find the container with id 123d2c924b9f85968d604347259f9e14263dc0754b98422afb8f0b2055aafdeb Nov 02 21:44:32.079604 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:32.074639 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9l4v9"] Nov 02 21:44:32.657568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:32.657534 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" event={"ID":"6730dff9-7d15-4a6a-bb30-63bccf531f4c","Type":"ContainerStarted","Data":"6c6503ecce2a367fef3c8c5a6888083d556962c136509acddade5d916c281f6d"} Nov 02 21:44:32.658907 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:32.658879 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" event={"ID":"34cdf2d8-2345-46fe-b006-fb6f05a86bf2","Type":"ContainerStarted","Data":"123d2c924b9f85968d604347259f9e14263dc0754b98422afb8f0b2055aafdeb"} Nov 02 21:44:35.675180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:35.675139 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" event={"ID":"34cdf2d8-2345-46fe-b006-fb6f05a86bf2","Type":"ContainerStarted","Data":"08d729f66aa8e2a9991641b1b68eeb9dc81297277c7a26098e00113e37a246f4"} Nov 02 21:44:35.675695 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:35.675204 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:35.676813 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:35.676788 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" event={"ID":"6730dff9-7d15-4a6a-bb30-63bccf531f4c","Type":"ContainerStarted","Data":"73f668e90e8dd6e29731bfaeb3c74045e19d82bcef59910fd9133b6fa73050d8"} Nov 02 21:44:35.676953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:35.676931 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:35.788164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:35.788096 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" podStartSLOduration=1.659886923 podStartE2EDuration="4.788083578s" totalImagesPullingTime="3.128196655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:35.786177719 +0000 UTC m=+614.747728039" watchObservedRunningTime="2026-11-02 21:44:35.788083578 +0000 UTC m=+614.749633898" Nov 02 21:44:35.900214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:35.900161 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" podStartSLOduration=1.98133428 podStartE2EDuration="4.900120534s" totalImagesPullingTime="2.918786254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:35.89894909 +0000 UTC m=+614.860499411" watchObservedRunningTime="2026-11-02 21:44:35.900120534 +0000 UTC m=+614.861670854" Nov 02 21:44:38.639257 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:38.639225 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:38.982740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:38.982707 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-648dd6fb45-v9bkq"] Nov 02 21:44:38.986324 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:38.986308 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.043676 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.043651 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-648dd6fb45-v9bkq"] Nov 02 21:44:39.101455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.101423 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c47b892-df35-4707-9e8e-45c0999acf87-metrics-certs\") pod \"ai-gateway-controller-648dd6fb45-v9bkq\" (UID: \"5c47b892-df35-4707-9e8e-45c0999acf87\") " pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.101455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.101458 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrxs\" (UniqueName: \"kubernetes.io/projected/5c47b892-df35-4707-9e8e-45c0999acf87-kube-api-access-xkrxs\") pod \"ai-gateway-controller-648dd6fb45-v9bkq\" (UID: \"5c47b892-df35-4707-9e8e-45c0999acf87\") " pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.202959 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.202929 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c47b892-df35-4707-9e8e-45c0999acf87-metrics-certs\") pod \"ai-gateway-controller-648dd6fb45-v9bkq\" (UID: \"5c47b892-df35-4707-9e8e-45c0999acf87\") " pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.202959 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.202961 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrxs\" (UniqueName: \"kubernetes.io/projected/5c47b892-df35-4707-9e8e-45c0999acf87-kube-api-access-xkrxs\") pod \"ai-gateway-controller-648dd6fb45-v9bkq\" (UID: \"5c47b892-df35-4707-9e8e-45c0999acf87\") " pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.205296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.205271 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c47b892-df35-4707-9e8e-45c0999acf87-metrics-certs\") pod \"ai-gateway-controller-648dd6fb45-v9bkq\" (UID: \"5c47b892-df35-4707-9e8e-45c0999acf87\") " pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.259475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.259411 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrxs\" (UniqueName: \"kubernetes.io/projected/5c47b892-df35-4707-9e8e-45c0999acf87-kube-api-access-xkrxs\") pod \"ai-gateway-controller-648dd6fb45-v9bkq\" (UID: \"5c47b892-df35-4707-9e8e-45c0999acf87\") " pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.298873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.298844 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:39.663582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.663560 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-648dd6fb45-v9bkq"] Nov 02 21:44:39.665591 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:44:39.665561 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c47b892_df35_4707_9e8e_45c0999acf87.slice/crio-848edd2133394391dfd9d73bbdeb1c4bb2a3c335c2c1401d3f98ba087a6aaa07 WatchSource:0}: Error finding container 848edd2133394391dfd9d73bbdeb1c4bb2a3c335c2c1401d3f98ba087a6aaa07: Status 404 returned error can't find the container with id 848edd2133394391dfd9d73bbdeb1c4bb2a3c335c2c1401d3f98ba087a6aaa07 Nov 02 21:44:39.701888 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:39.701858 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" event={"ID":"5c47b892-df35-4707-9e8e-45c0999acf87","Type":"ContainerStarted","Data":"848edd2133394391dfd9d73bbdeb1c4bb2a3c335c2c1401d3f98ba087a6aaa07"} Nov 02 21:44:40.707968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:40.707885 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" event={"ID":"5c47b892-df35-4707-9e8e-45c0999acf87","Type":"ContainerStarted","Data":"7363bb1c160354e2a17b823a0dc76540171ce38502ec1a8a32941cb3f6893ce1"} Nov 02 21:44:40.708373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:40.708014 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:40.821162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:40.821092 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" podStartSLOduration=2.033176361 podStartE2EDuration="2.821078257s" totalImagesPullingTime="787.901896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:40.820104102 +0000 UTC m=+619.781654422" watchObservedRunningTime="2026-11-02 21:44:40.821078257 +0000 UTC m=+619.782628608" Nov 02 21:44:46.004790 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:44:46.004754 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:44:46.690621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:46.690592 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rdz8f" Nov 02 21:44:46.690804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:46.690657 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9l4v9" Nov 02 21:44:51.714339 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:51.714303 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-648dd6fb45-v9bkq" Nov 02 21:44:51.877909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:51.877876 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-qc7ww"] Nov 02 21:44:51.878173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:51.878145 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" podUID="320cdede-8be0-4443-bd6a-640757afe583" containerName="manager" containerID="cri-o://f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46" gracePeriod=30 Nov 02 21:44:52.125063 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.125040 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:52.216003 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.215971 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58\" (UniqueName: \"kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58\") pod \"320cdede-8be0-4443-bd6a-640757afe583\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " Nov 02 21:44:52.216174 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.216045 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs\" (UniqueName: \"kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs\") pod \"320cdede-8be0-4443-bd6a-640757afe583\" (UID: \"320cdede-8be0-4443-bd6a-640757afe583\") " Nov 02 21:44:52.218158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.218110 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58" pod "320cdede-8be0-4443-bd6a-640757afe583" (UID: "320cdede-8be0-4443-bd6a-640757afe583"). InnerVolumeSpecName "kube-api-access-k4f58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:52.218243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.218120 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs" pod "320cdede-8be0-4443-bd6a-640757afe583" (UID: "320cdede-8be0-4443-bd6a-640757afe583"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:52.317225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.317118 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4f58\" (UniqueName: \"kubernetes.io/projected/320cdede-8be0-4443-bd6a-640757afe583-kube-api-access-k4f58\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:44:52.317225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.317176 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320cdede-8be0-4443-bd6a-640757afe583-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:44:52.761658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.761618 4285 generic.go:356] "Generic (PLEG): container finished" podID="320cdede-8be0-4443-bd6a-640757afe583" containerID="f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46" exitCode=0 Nov 02 21:44:52.761658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.761653 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" event={"ID":"320cdede-8be0-4443-bd6a-640757afe583","Type":"ContainerDied","Data":"f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46"} Nov 02 21:44:52.762141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.761668 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" Nov 02 21:44:52.762141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.761707 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c7987c6b6-qc7ww" event={"ID":"320cdede-8be0-4443-bd6a-640757afe583","Type":"ContainerDied","Data":"456f0e53e1904502d7356b3ebd90e55648d8172a5717f43b9283ddb2183268b4"} Nov 02 21:44:52.762141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.761730 4285 scope.go:122] "RemoveContainer" containerID="f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46" Nov 02 21:44:52.771000 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.770983 4285 scope.go:122] "RemoveContainer" containerID="f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46" Nov 02 21:44:52.771292 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:44:52.771269 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46\": container with ID starting with f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46 not found: ID does not exist" containerID="f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46" Nov 02 21:44:52.771359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.771304 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46"} err="failed to get container status \"f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46\": rpc error: code = NotFound desc = could not find container \"f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46\": container with ID starting with f75332b2a1d433aec1467bc823710f21784bb78bfb3bd860b14f518c00a3cc46 not found: ID does not exist" Nov 02 21:44:52.923832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.923794 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-qc7ww"] Nov 02 21:44:52.973295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:52.973232 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-c7987c6b6-qc7ww"] Nov 02 21:44:53.171621 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:53.171537 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320cdede-8be0-4443-bd6a-640757afe583" path="/var/lib/kubelet/pods/320cdede-8be0-4443-bd6a-640757afe583/volumes" Nov 02 21:44:53.427792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:53.427713 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:53.902829 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:53.902798 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:55.399398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:55.399357 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:55.565390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:55.565355 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:56.976217 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:56.976184 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:57.137542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:44:57.137511 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:00.164921 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.164885 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj"] Nov 02 21:45:00.165349 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.165290 4285 cpu_manager.go:414] "containerMap: removing container" podUID="320cdede-8be0-4443-bd6a-640757afe583" containerName="manager" Nov 02 21:45:00.165349 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.165302 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320cdede-8be0-4443-bd6a-640757afe583" containerName="manager" Nov 02 21:45:00.165418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.165385 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="320cdede-8be0-4443-bd6a-640757afe583" containerName="manager" Nov 02 21:45:00.170831 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.170798 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.224832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.224798 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj"] Nov 02 21:45:00.271819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.271785 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:45:00.292909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.292864 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-vhhfj\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.293118 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.292923 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqh7p\" (UniqueName: \"kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p\") pod \"maas-api-key-cleanup-29894265-vhhfj\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.393694 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.393646 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-vhhfj\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.393876 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.393706 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqh7p\" (UniqueName: \"kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p\") pod \"maas-api-key-cleanup-29894265-vhhfj\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.394300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.394277 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894265-vhhfj\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.451063 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.451022 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqh7p\" (UniqueName: \"kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p\") pod \"maas-api-key-cleanup-29894265-vhhfj\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.484639 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.484602 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:00.857775 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.857750 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj"] Nov 02 21:45:00.859578 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:00.859551 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b35febd_ece5_476c_a13c_5f92df40f6f4.slice/crio-a4ab981838f6c0734baacfcfcd2f594160d3dd3227924b29396079b934797aa9 WatchSource:0}: Error finding container a4ab981838f6c0734baacfcfcd2f594160d3dd3227924b29396079b934797aa9: Status 404 returned error can't find the container with id a4ab981838f6c0734baacfcfcd2f594160d3dd3227924b29396079b934797aa9 Nov 02 21:45:00.861425 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:00.861404 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:45:01.805705 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:01.805666 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerStarted","Data":"a4ab981838f6c0734baacfcfcd2f594160d3dd3227924b29396079b934797aa9"} Nov 02 21:45:02.178647 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:02.178563 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:04.069820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.069742 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l"] Nov 02 21:45:04.073582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.073561 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.119449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.119417 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl"] Nov 02 21:45:04.123019 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.123000 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.176661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.176632 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l"] Nov 02 21:45:04.227737 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.227709 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl"] Nov 02 21:45:04.230546 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230518 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dck\" (UniqueName: \"kubernetes.io/projected/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-kube-api-access-27dck\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230561 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230598 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230623 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230661 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230698 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230732 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230792 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230808 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230848 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230868 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230886 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.230922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230905 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmf8\" (UniqueName: \"kubernetes.io/projected/cd208f3f-912d-4600-982d-22c0ff27389e-kube-api-access-mmmf8\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.231200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230933 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd208f3f-912d-4600-982d-22c0ff27389e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.231200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230954 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.231200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230974 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd208f3f-912d-4600-982d-22c0ff27389e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.231200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.230990 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd208f3f-912d-4600-982d-22c0ff27389e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.231200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.231039 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.331974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.331894 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dck\" (UniqueName: \"kubernetes.io/projected/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-kube-api-access-27dck\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.331974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.331933 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.331974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.331955 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332160 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332215 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.332370 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332259 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332370 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332294 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.332370 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332352 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.332521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332378 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332393 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332493 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332603 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.332685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332651 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332677 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.332791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332684 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332741 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.332791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332777 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmf8\" (UniqueName: \"kubernetes.io/projected/cd208f3f-912d-4600-982d-22c0ff27389e-kube-api-access-mmmf8\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332807 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd208f3f-912d-4600-982d-22c0ff27389e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332843 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332871 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd208f3f-912d-4600-982d-22c0ff27389e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332898 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd208f3f-912d-4600-982d-22c0ff27389e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332932 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.333041 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332991 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333338 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.333319 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333374 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.332740 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.333374 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.333351 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.333507 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.333487 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.333628 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.333611 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd208f3f-912d-4600-982d-22c0ff27389e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.334799 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.334775 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.335642 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.335620 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd208f3f-912d-4600-982d-22c0ff27389e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.335838 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.335816 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.336089 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.336066 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd208f3f-912d-4600-982d-22c0ff27389e-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.388332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.388300 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dck\" (UniqueName: \"kubernetes.io/projected/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-kube-api-access-27dck\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.388811 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.388777 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb34991-dfd5-4e0f-9639-0aad1f6f4447-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-blcrl\" (UID: \"6eb34991-dfd5-4e0f-9639-0aad1f6f4447\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.389081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.389063 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd208f3f-912d-4600-982d-22c0ff27389e-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.389835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.389811 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmf8\" (UniqueName: \"kubernetes.io/projected/cd208f3f-912d-4600-982d-22c0ff27389e-kube-api-access-mmmf8\") pod \"maas-default-gateway-openshift-default-65c448b547-mcx5l\" (UID: \"cd208f3f-912d-4600-982d-22c0ff27389e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.437343 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.437311 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:04.689605 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.689529 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:04.868069 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.868046 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl"] Nov 02 21:45:04.870469 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:04.870439 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb34991_dfd5_4e0f_9639_0aad1f6f4447.slice/crio-efabecd55435bd643a5e42c46f2d38a0101a2e4edb0054e93fff0899c1ee8e03 WatchSource:0}: Error finding container efabecd55435bd643a5e42c46f2d38a0101a2e4edb0054e93fff0899c1ee8e03: Status 404 returned error can't find the container with id efabecd55435bd643a5e42c46f2d38a0101a2e4edb0054e93fff0899c1ee8e03 Nov 02 21:45:04.872969 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.872930 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.873081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.873007 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.873081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:04.873044 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:05.073927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.073896 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l"] Nov 02 21:45:05.076148 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:05.076087 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd208f3f_912d_4600_982d_22c0ff27389e.slice/crio-6873108c8320466c5f9d511fd2e27766106a01b563a3906e6c3b5704a0041195 WatchSource:0}: Error finding container 6873108c8320466c5f9d511fd2e27766106a01b563a3906e6c3b5704a0041195: Status 404 returned error can't find the container with id 6873108c8320466c5f9d511fd2e27766106a01b563a3906e6c3b5704a0041195 Nov 02 21:45:05.078649 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.078615 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:05.078758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.078681 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:05.078758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.078708 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:05.804491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.804409 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:05.861342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.861304 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerStarted","Data":"0ad2c6eee464548fe4c00cc270818d8be6ab79b087d2ec83501737f38ef9914d"} Nov 02 21:45:05.866167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.864925 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" event={"ID":"cd208f3f-912d-4600-982d-22c0ff27389e","Type":"ContainerStarted","Data":"9f338e9f97df55cd58eeb97baadb608cd5d5c285bf50d2e942e7bcf1775e90f3"} Nov 02 21:45:05.866167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.864976 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" event={"ID":"cd208f3f-912d-4600-982d-22c0ff27389e","Type":"ContainerStarted","Data":"6873108c8320466c5f9d511fd2e27766106a01b563a3906e6c3b5704a0041195"} Nov 02 21:45:05.867162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.866674 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" event={"ID":"6eb34991-dfd5-4e0f-9639-0aad1f6f4447","Type":"ContainerStarted","Data":"9fd9f035dbf5f132ae83a9e78dc5795e13d43f4b1aa1ae1f4bb46543f5a8db0a"} Nov 02 21:45:05.867162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.866734 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" event={"ID":"6eb34991-dfd5-4e0f-9639-0aad1f6f4447","Type":"ContainerStarted","Data":"efabecd55435bd643a5e42c46f2d38a0101a2e4edb0054e93fff0899c1ee8e03"} Nov 02 21:45:05.991467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:05.991418 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" podStartSLOduration=1.6026773479999998 podStartE2EDuration="5.99140464s" totalImagesPullingTime="4.388727292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:05.98907926 +0000 UTC m=+644.950629581" watchObservedRunningTime="2026-11-02 21:45:05.99140464 +0000 UTC m=+644.952954961" Nov 02 21:45:06.132207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.116461 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podStartSLOduration=2.116445422 podStartE2EDuration="2.116445422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:06.11418373 +0000 UTC m=+645.075734050" watchObservedRunningTime="2026-11-02 21:45:06.116445422 +0000 UTC m=+645.077995743" Nov 02 21:45:06.462207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.453014 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:06.462207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.455236 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:06.462207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.455289 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:06.690274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.690248 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:06.691690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.691663 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:45:06.691809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.691722 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 21:45:06.872911 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.872828 4285 generic.go:356] "Generic (PLEG): container finished" podID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerID="0ad2c6eee464548fe4c00cc270818d8be6ab79b087d2ec83501737f38ef9914d" exitCode=60 Nov 02 21:45:06.873053 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.872918 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerDied","Data":"0ad2c6eee464548fe4c00cc270818d8be6ab79b087d2ec83501737f38ef9914d"} Nov 02 21:45:06.873104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.873059 4285 scope.go:122] "RemoveContainer" containerID="0ad2c6eee464548fe4c00cc270818d8be6ab79b087d2ec83501737f38ef9914d" Nov 02 21:45:06.986417 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:06.986367 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podStartSLOduration=2.986353692 podStartE2EDuration="2.986353692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:06.234944441 +0000 UTC m=+645.196494764" watchObservedRunningTime="2026-11-02 21:45:06.986353692 +0000 UTC m=+645.947904012" Nov 02 21:45:07.443202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.441220 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:07.443202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.441279 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:07.690190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.690155 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:45:07.690372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.690208 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 21:45:07.878502 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.878471 4285 generic.go:356] "Generic (PLEG): container finished" podID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerID="d8115b5f67016041ab93c135542bc2522397b60d5b493f5e8349ca8103efc492" exitCode=60 Nov 02 21:45:07.878662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.878558 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerDied","Data":"d8115b5f67016041ab93c135542bc2522397b60d5b493f5e8349ca8103efc492"} Nov 02 21:45:07.878662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.878617 4285 scope.go:122] "RemoveContainer" containerID="0ad2c6eee464548fe4c00cc270818d8be6ab79b087d2ec83501737f38ef9914d" Nov 02 21:45:07.879084 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:07.879065 4285 scope.go:122] "RemoveContainer" containerID="d8115b5f67016041ab93c135542bc2522397b60d5b493f5e8349ca8103efc492" Nov 02 21:45:07.879303 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:07.879281 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-vhhfj_odh-ai-gateway-infra(2b35febd-ece5-476c-a13c-5f92df40f6f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" Nov 02 21:45:08.442147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:08.441221 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:08.442147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:08.441282 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:08.690598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:08.690553 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:45:08.690980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:08.690616 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 21:45:08.885161 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:08.885072 4285 scope.go:122] "RemoveContainer" containerID="d8115b5f67016041ab93c135542bc2522397b60d5b493f5e8349ca8103efc492" Nov 02 21:45:08.885391 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:08.885371 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-vhhfj_odh-ai-gateway-infra(2b35febd-ece5-476c-a13c-5f92df40f6f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" Nov 02 21:45:09.439148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:09.438228 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:09.439148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:09.438289 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:09.690058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:09.689975 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:45:09.690058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:09.690027 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 21:45:10.448150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:10.444238 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:10.448150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:10.444296 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:10.690714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:10.690681 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:45:10.690951 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:10.690929 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 21:45:11.437743 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:11.437678 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:11.437743 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:11.437735 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:11.689944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:11.689848 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:45:11.689944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:11.689897 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 21:45:12.437591 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:12.437559 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:45:12.437796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:12.437614 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 02 21:45:13.413975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:13.413943 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:13.414660 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:13.414260 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:13.415047 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:13.415018 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" Nov 02 21:45:13.441102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:13.441082 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:13.441373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:13.441349 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:13.442158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:13.442142 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" Nov 02 21:45:16.003856 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:16.003817 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:45:21.714253 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:21.714226 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:21.866252 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:21.866219 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:22.166114 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.166045 4285 scope.go:122] "RemoveContainer" containerID="d8115b5f67016041ab93c135542bc2522397b60d5b493f5e8349ca8103efc492" Nov 02 21:45:22.201692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.201665 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj"] Nov 02 21:45:22.206092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.206067 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.299501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.299477 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb8112-openshift-default-dockercfg-r4kpc\"" Nov 02 21:45:22.310112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.310083 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj"] Nov 02 21:45:22.312878 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.312853 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.312891 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.312923 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.312951 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.313002 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.313049 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.313105 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.313163 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zxg\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.313283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.313223 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.321887 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.321839 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:22.414391 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414360 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414551 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414409 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414551 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414443 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414683 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414541 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zxg\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414683 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414612 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414683 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414671 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414698 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414726 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414755 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.414837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.414779 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.415116 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.415094 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.415209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.415187 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.415263 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.415245 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.415335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.415313 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.417200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.417097 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.417555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.417446 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.476574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.476543 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zxg\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.476717 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.476693 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.514055 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.514024 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb"] Nov 02 21:45:22.517568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.517552 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.517877 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.517853 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:22.570508 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.569888 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-64438f-openshift-default-dockercfg-tx22g\"" Nov 02 21:45:22.585258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.585189 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb"] Nov 02 21:45:22.595678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.595632 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616843 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616885 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616909 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616928 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616947 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzjm\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616976 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.616999 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.617088 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.617544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.617113 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718007 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718048 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718167 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718209 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718226 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzjm\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718250 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718273 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718400 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718446 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718526 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718583 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718759 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718821 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.718925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.718866 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.720674 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.720657 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.720898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.720883 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.751258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.751233 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:22.775171 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.775116 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.775261 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.775194 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzjm\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.844030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.844003 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:22.916021 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.915993 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:22.957735 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.957706 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerStarted","Data":"7442b438c3ce1113ac6d89e5e1bb42e6b4cd4d8c0172f79e2a213a9703ef95ba"} Nov 02 21:45:22.986942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.986916 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj"] Nov 02 21:45:22.997723 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.997687 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:22.997807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.997761 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:22.997807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:22.997797 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:23.076330 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.076300 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.232093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.232050 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb"] Nov 02 21:45:23.253145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.246762 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:23.253145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.246867 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:23.253145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.246930 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:23.257084 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.257062 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.467197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.467166 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx"] Nov 02 21:45:23.467386 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.467209 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.471115 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.471093 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.543630 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.543549 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.543916 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.543901 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544039 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544024 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544149 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544257 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544364 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544458 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544560 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544549 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.544659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.544647 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl6kz\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.546032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.546007 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx"] Nov 02 21:45:23.582643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.582616 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-10fdd5-openshift-default-dockercfg-pg4pr\"" Nov 02 21:45:23.630490 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.625492 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.645547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645510 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.645750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645584 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.645750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645626 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.645750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645651 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.645750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645696 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.645750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645724 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645752 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645782 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6kz\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.645866 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646386 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.646365 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646479 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.646447 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646619 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.646599 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.646837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.646814 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.647229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.647203 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.649111 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.649083 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.649270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.649249 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.702551 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.702513 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.703205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.703183 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6kz\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.784291 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.784257 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:23.785795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.785773 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj"] Nov 02 21:45:23.789059 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.789044 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.841351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.841320 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ffe386-openshift-default-dockercfg-8xxrb\"" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849658 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9jf\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849714 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849743 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849791 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849886 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849911 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849937 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.849965 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.850002 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.854061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.850097 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj"] Nov 02 21:45:23.891431 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.891405 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951300 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951350 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951379 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951409 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951451 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951538 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9jf\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951582 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951611 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.951661 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.952313 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.952835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.952557 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.961865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.959584 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.961865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.959846 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.961865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.960072 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.964604 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.964579 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.964762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.964648 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:23.969480 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.969267 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" event={"ID":"acb34325-144e-449c-be36-7240ee0f4acd","Type":"ContainerStarted","Data":"82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54"} Nov 02 21:45:23.969480 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.969317 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" event={"ID":"acb34325-144e-449c-be36-7240ee0f4acd","Type":"ContainerStarted","Data":"525a84f5fa62cbc818e61c3d72b2ffa33400243df793f8d80c4f9d8bbef917b6"} Nov 02 21:45:23.970810 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.970784 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" event={"ID":"700c6c8d-fb45-4029-9204-ad3e41f0657a","Type":"ContainerStarted","Data":"3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b"} Nov 02 21:45:23.970901 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.970831 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" event={"ID":"700c6c8d-fb45-4029-9204-ad3e41f0657a","Type":"ContainerStarted","Data":"68a60343f0f5da1dfa7a865444aa8bc2b5ed01e55368abb282be766d520ccff5"} Nov 02 21:45:23.972482 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.972463 4285 generic.go:356] "Generic (PLEG): container finished" podID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerID="7442b438c3ce1113ac6d89e5e1bb42e6b4cd4d8c0172f79e2a213a9703ef95ba" exitCode=60 Nov 02 21:45:23.972568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.972533 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerDied","Data":"7442b438c3ce1113ac6d89e5e1bb42e6b4cd4d8c0172f79e2a213a9703ef95ba"} Nov 02 21:45:23.972612 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.972582 4285 scope.go:122] "RemoveContainer" containerID="d8115b5f67016041ab93c135542bc2522397b60d5b493f5e8349ca8103efc492" Nov 02 21:45:23.972909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:23.972894 4285 scope.go:122] "RemoveContainer" containerID="7442b438c3ce1113ac6d89e5e1bb42e6b4cd4d8c0172f79e2a213a9703ef95ba" Nov 02 21:45:23.973118 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:23.973101 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894265-vhhfj_odh-ai-gateway-infra(2b35febd-ece5-476c-a13c-5f92df40f6f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" Nov 02 21:45:24.015001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.014981 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:24.015104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.015090 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9jf\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:24.040146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.040107 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:24.081012 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.080946 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj"] Nov 02 21:45:24.085719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.085679 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podStartSLOduration=2.08566723 podStartE2EDuration="2.08566723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:24.084814683 +0000 UTC m=+663.046365003" watchObservedRunningTime="2026-11-02 21:45:24.08566723 +0000 UTC m=+663.047217549" Nov 02 21:45:24.103409 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.103388 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:24.164416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.164342 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx"] Nov 02 21:45:24.181652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.181616 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:24.181747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.181700 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:24.181796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.181745 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:24.226229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.226200 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:24.319485 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.317915 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podStartSLOduration=2.317899936 podStartE2EDuration="2.317899936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:24.316119134 +0000 UTC m=+663.277669456" watchObservedRunningTime="2026-11-02 21:45:24.317899936 +0000 UTC m=+663.279450256" Nov 02 21:45:24.386796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.386717 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:24.485880 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.485854 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj"] Nov 02 21:45:24.488621 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:24.488588 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e29ece0_7361_4bae_8651_18356ed45965.slice/crio-679bda35659e4b7b9db77d03f9942817e458c2084fb5b518e06434f1434d3ad0 WatchSource:0}: Error finding container 679bda35659e4b7b9db77d03f9942817e458c2084fb5b518e06434f1434d3ad0: Status 404 returned error can't find the container with id 679bda35659e4b7b9db77d03f9942817e458c2084fb5b518e06434f1434d3ad0 Nov 02 21:45:24.491198 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.491165 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:24.491306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.491234 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:24.491306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.491267 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:24.518841 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.518815 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:24.524965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.524946 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:24.844979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.844945 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:24.850773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.850743 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:24.981096 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.981064 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" event={"ID":"9e29ece0-7361-4bae-8651-18356ed45965","Type":"ContainerStarted","Data":"2740ef9f1a718fc9c17d7684fcfedc15c552b8c307d5bc9510bc4008eb18f940"} Nov 02 21:45:24.981259 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.981152 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" event={"ID":"9e29ece0-7361-4bae-8651-18356ed45965","Type":"ContainerStarted","Data":"679bda35659e4b7b9db77d03f9942817e458c2084fb5b518e06434f1434d3ad0"} Nov 02 21:45:24.982635 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.982608 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" event={"ID":"08ab0a7c-02dc-4187-8d69-5f4fe48cb164","Type":"ContainerStarted","Data":"f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0"} Nov 02 21:45:24.982765 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.982671 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:24.982765 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.982740 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:24.982874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.982782 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" event={"ID":"08ab0a7c-02dc-4187-8d69-5f4fe48cb164","Type":"ContainerStarted","Data":"646f539871df381cd7227555e09ac9e0b245789d03bf95853832a68f03ee2d46"} Nov 02 21:45:24.984139 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.984107 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:45:24.984233 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:24.984214 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:25.103850 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.103822 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:25.108991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.108969 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:25.113818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.113800 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:25.165031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.165001 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p\" (UniqueName: \"kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p\") pod \"2b35febd-ece5-476c-a13c-5f92df40f6f4\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " Nov 02 21:45:25.165183 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.165163 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca\") pod \"2b35febd-ece5-476c-a13c-5f92df40f6f4\" (UID: \"2b35febd-ece5-476c-a13c-5f92df40f6f4\") " Nov 02 21:45:25.165462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.165439 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca" pod "2b35febd-ece5-476c-a13c-5f92df40f6f4" (UID: "2b35febd-ece5-476c-a13c-5f92df40f6f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:25.165537 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.165514 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b35febd-ece5-476c-a13c-5f92df40f6f4-maas-api-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:45:25.167172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.167145 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p" pod "2b35febd-ece5-476c-a13c-5f92df40f6f4" (UID: "2b35febd-ece5-476c-a13c-5f92df40f6f4"). InnerVolumeSpecName "kube-api-access-dqh7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:25.266840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.266807 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqh7p\" (UniqueName: \"kubernetes.io/projected/2b35febd-ece5-476c-a13c-5f92df40f6f4-kube-api-access-dqh7p\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:45:25.439271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.439175 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podStartSLOduration=2.439161024 podStartE2EDuration="2.439161024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:25.437437369 +0000 UTC m=+664.398987702" watchObservedRunningTime="2026-11-02 21:45:25.439161024 +0000 UTC m=+664.400711344" Nov 02 21:45:25.552760 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.552710 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podStartSLOduration=2.552695152 podStartE2EDuration="2.552695152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:25.551002661 +0000 UTC m=+664.512552981" watchObservedRunningTime="2026-11-02 21:45:25.552695152 +0000 UTC m=+664.514245472" Nov 02 21:45:25.784878 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.784844 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:25.789928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.789904 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:25.988465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.988437 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" Nov 02 21:45:25.988465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.988429 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj" event={"ID":"2b35febd-ece5-476c-a13c-5f92df40f6f4","Type":"ContainerDied","Data":"a4ab981838f6c0734baacfcfcd2f594160d3dd3227924b29396079b934797aa9"} Nov 02 21:45:25.988968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.988518 4285 scope.go:122] "RemoveContainer" containerID="7442b438c3ce1113ac6d89e5e1bb42e6b4cd4d8c0172f79e2a213a9703ef95ba" Nov 02 21:45:25.988968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.988747 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:25.989457 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.989436 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:25.989947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.989890 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:45:25.990231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:25.990213 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:45:26.378704 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:26.378671 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj"] Nov 02 21:45:26.386655 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:26.386627 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894265-vhhfj"] Nov 02 21:45:27.129668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:27.129636 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:27.174053 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:27.174019 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" path="/var/lib/kubelet/pods/2b35febd-ece5-476c-a13c-5f92df40f6f4/volumes" Nov 02 21:45:27.302133 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:27.302085 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:27.476487 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:27.476449 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:28.086917 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:28.086873 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:28.244431 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:28.244401 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:28.422445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:28.422364 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:28.823506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:28.823470 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:28.986141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:28.986086 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:29.369482 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:29.369451 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:29.547393 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:29.547361 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:29.703884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:29.703853 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.043404 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.043373 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.222540 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.222512 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.381930 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.381848 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.561728 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.561687 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.719310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.719274 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.977818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.977734 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:31.978323 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978302 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd"] Nov 02 21:45:31.978743 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978729 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978745 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978758 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978767 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978777 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978782 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978863 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.978925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.978871 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:31.983101 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:31.983084 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.040076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.039614 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd"] Nov 02 21:45:32.051141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.050765 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvsk4\" (UniqueName: \"kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4\") pod \"payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.051141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.050824 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.086883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.086785 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-64438f-dockercfg-hrx4n\"" Nov 02 21:45:32.086883 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.086852 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-64438f\"" Nov 02 21:45:32.152474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.152434 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvsk4\" (UniqueName: \"kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4\") pod \"payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.152662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.152488 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.153362 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.153330 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.209224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.209196 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvsk4\" (UniqueName: \"kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4\") pod \"payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.296013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.295942 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:32.664001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:32.663975 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd"] Nov 02 21:45:32.666480 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:32.666451 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec39698b_c904_4283_86d5_e6b9b5638f2d.slice/crio-0a1bcd56582ea09651e2f5bf7eef9448e37233808834e5d048d4d3f36ae4b07c WatchSource:0}: Error finding container 0a1bcd56582ea09651e2f5bf7eef9448e37233808834e5d048d4d3f36ae4b07c: Status 404 returned error can't find the container with id 0a1bcd56582ea09651e2f5bf7eef9448e37233808834e5d048d4d3f36ae4b07c Nov 02 21:45:33.025719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.025674 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" event={"ID":"ec39698b-c904-4283-86d5-e6b9b5638f2d","Type":"ContainerStarted","Data":"45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910"} Nov 02 21:45:33.025896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.025740 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:33.025896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.025765 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" event={"ID":"ec39698b-c904-4283-86d5-e6b9b5638f2d","Type":"ContainerStarted","Data":"0a1bcd56582ea09651e2f5bf7eef9448e37233808834e5d048d4d3f36ae4b07c"} Nov 02 21:45:33.139162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.139091 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" podStartSLOduration=2.139074683 podStartE2EDuration="2.139074683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:33.137778053 +0000 UTC m=+672.099328374" watchObservedRunningTime="2026-11-02 21:45:33.139074683 +0000 UTC m=+672.100625033" Nov 02 21:45:33.555671 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.555564 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:33.704845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.704810 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:33.990085 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:33.990055 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:36.337781 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.337734 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:36.501567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.501537 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:36.882571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.882534 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7"] Nov 02 21:45:36.883296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.883280 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b35febd-ece5-476c-a13c-5f92df40f6f4" containerName="cleanup" Nov 02 21:45:36.886613 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.886593 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:36.942987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.942949 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7"] Nov 02 21:45:36.988933 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.988896 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fb8112\"" Nov 02 21:45:36.989168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:36.989030 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-fb8112-dockercfg-kpfmk\"" Nov 02 21:45:37.040486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.040452 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7lk\" (UniqueName: \"kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk\") pod \"payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.040657 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.040520 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.141561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.141477 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7lk\" (UniqueName: \"kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk\") pod \"payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.141561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.141543 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.142264 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.142246 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.197682 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.197654 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7lk\" (UniqueName: \"kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk\") pod \"payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.200476 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.200441 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:37.569474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.569442 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7"] Nov 02 21:45:37.704871 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.704836 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:37.877413 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:37.877327 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:38.050939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.050896 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" event={"ID":"9c7a564d-5b31-49be-8198-2f8103208df1","Type":"ContainerStarted","Data":"02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa"} Nov 02 21:45:38.051098 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.050996 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" event={"ID":"9c7a564d-5b31-49be-8198-2f8103208df1","Type":"ContainerStarted","Data":"2a1a1c1f3db14940270aa8f00405d3c8fa1a12ec9a3d1bfb6b62e00d334c328f"} Nov 02 21:45:38.051098 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.051029 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:38.164961 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.164850 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" podStartSLOduration=2.164836043 podStartE2EDuration="2.164836043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:38.162196112 +0000 UTC m=+677.123746431" watchObservedRunningTime="2026-11-02 21:45:38.164836043 +0000 UTC m=+677.126386364" Nov 02 21:45:38.478275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.478241 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg"] Nov 02 21:45:38.482472 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.482453 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.536494 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.536467 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9cc56b-serving-cert\"" Nov 02 21:45:38.552066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.552037 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4hn\" (UniqueName: \"kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.552214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.552147 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.552272 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.552257 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.552314 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.552292 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.585687 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.585659 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cbb6p\"" Nov 02 21:45:38.586158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.585769 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:45:38.588466 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.588444 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg"] Nov 02 21:45:38.653489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.653455 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4hn\" (UniqueName: \"kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.653640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.653522 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.653640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.653626 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.653742 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.653674 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.653742 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:38.653678 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-serving-cert: secret "maas-api-e2e-worker-w2-9cc56b-serving-cert" not found Nov 02 21:45:38.653859 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:38.653749 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls podName:bbdb6075-2020-48d9-8fdd-82e4285a9eca nodeName:}" failed. No retries permitted until 2026-11-02 21:45:39.153732001 +0000 UTC m=+678.115282311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls") pod "maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" (UID: "bbdb6075-2020-48d9-8fdd-82e4285a9eca") : secret "maas-api-e2e-worker-w2-9cc56b-serving-cert" not found Nov 02 21:45:38.654378 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.654347 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.655974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.655956 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.693380 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.693350 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:38.709870 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.709837 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4hn\" (UniqueName: \"kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:38.848372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:38.848294 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:39.005317 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:39.005236 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:39.158714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:39.158627 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:39.161059 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:39.161031 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:39.395921 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:39.395885 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:39.766938 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:39.766858 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg"] Nov 02 21:45:39.771063 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:39.771030 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdb6075_2020_48d9_8fdd_82e4285a9eca.slice/crio-a509daedb96e046776449d1133ff045ccbf45bcf49222fe24736b08b120b7a34 WatchSource:0}: Error finding container a509daedb96e046776449d1133ff045ccbf45bcf49222fe24736b08b120b7a34: Status 404 returned error can't find the container with id a509daedb96e046776449d1133ff045ccbf45bcf49222fe24736b08b120b7a34 Nov 02 21:45:40.027698 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:40.027608 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:40.063780 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:40.063743 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" event={"ID":"bbdb6075-2020-48d9-8fdd-82e4285a9eca","Type":"ContainerStarted","Data":"a509daedb96e046776449d1133ff045ccbf45bcf49222fe24736b08b120b7a34"} Nov 02 21:45:40.205088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:40.205040 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:40.369353 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:40.369268 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:41.070176 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:41.070110 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" event={"ID":"bbdb6075-2020-48d9-8fdd-82e4285a9eca","Type":"ContainerStarted","Data":"205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e"} Nov 02 21:45:41.070613 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:41.070207 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:41.187040 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:41.186984 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" podStartSLOduration=1.999256699 podStartE2EDuration="3.186971124s" totalImagesPullingTime="1.187714425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:41.186577101 +0000 UTC m=+680.148127422" watchObservedRunningTime="2026-11-02 21:45:41.186971124 +0000 UTC m=+680.148521444" Nov 02 21:45:43.493582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.493549 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:43.667916 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.667887 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:43.887304 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.887196 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh"] Nov 02 21:45:43.891792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.891764 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:43.917928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.917888 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqf9r\" (UniqueName: \"kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r\") pod \"payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:43.933250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.918176 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:43.943452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:43.943425 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f48c4d-dockercfg-p4nwn\"" Nov 02 21:45:44.005660 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.005619 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f48c4d\"" Nov 02 21:45:44.008284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.008254 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh"] Nov 02 21:45:44.046887 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.046850 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqf9r\" (UniqueName: \"kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r\") pod \"payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:44.047212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.047021 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft"] Nov 02 21:45:44.047475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.047292 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:44.048196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.048157 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:44.051832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.051816 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 21:45:44.052241 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.052200 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.091785 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.091754 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:44.102769 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.102740 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft"] Nov 02 21:45:44.105031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.105003 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqf9r\" (UniqueName: \"kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r\") pod \"payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:44.182217 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.168805 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9gf\" (UniqueName: \"kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf\") pod \"payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.182217 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.169075 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.206330 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.206298 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:44.260634 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.260594 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:44.269861 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.269828 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.270036 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.269874 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9gf\" (UniqueName: \"kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf\") pod \"payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.270604 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.270578 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.373086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.373001 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9gf\" (UniqueName: \"kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf\") pod \"payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:44.406833 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.406801 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:44.636489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.635017 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh"] Nov 02 21:45:44.639800 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:44.639769 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833c8ff0_54b9_4604_8af5_0832ffdc86d9.slice/crio-7a96c0037d690f67cb2b6a9204812ff184d4b42e2f03d8836be7393957999139 WatchSource:0}: Error finding container 7a96c0037d690f67cb2b6a9204812ff184d4b42e2f03d8836be7393957999139: Status 404 returned error can't find the container with id 7a96c0037d690f67cb2b6a9204812ff184d4b42e2f03d8836be7393957999139 Nov 02 21:45:44.672104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:44.672072 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:45.072009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:45.071976 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft"] Nov 02 21:45:45.073586 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:45:45.073558 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c858eb_093f_4e85_8338_094bb6743dfa.slice/crio-45a843a8589a0e7df71ce2b1a6c79acc2884c259888b98dee2404be925734d2f WatchSource:0}: Error finding container 45a843a8589a0e7df71ce2b1a6c79acc2884c259888b98dee2404be925734d2f: Status 404 returned error can't find the container with id 45a843a8589a0e7df71ce2b1a6c79acc2884c259888b98dee2404be925734d2f Nov 02 21:45:45.090455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:45.090420 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" event={"ID":"833c8ff0-54b9-4604-8af5-0832ffdc86d9","Type":"ContainerStarted","Data":"9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb"} Nov 02 21:45:45.090600 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:45.090482 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" event={"ID":"833c8ff0-54b9-4604-8af5-0832ffdc86d9","Type":"ContainerStarted","Data":"7a96c0037d690f67cb2b6a9204812ff184d4b42e2f03d8836be7393957999139"} Nov 02 21:45:45.090676 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:45.090629 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:45.091809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:45.091784 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" event={"ID":"e9c858eb-093f-4e85-8338-094bb6743dfa","Type":"ContainerStarted","Data":"45a843a8589a0e7df71ce2b1a6c79acc2884c259888b98dee2404be925734d2f"} Nov 02 21:45:45.204507 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:45.204409 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" podStartSLOduration=2.204395662 podStartE2EDuration="2.204395662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:45.202111173 +0000 UTC m=+684.163661493" watchObservedRunningTime="2026-11-02 21:45:45.204395662 +0000 UTC m=+684.165945980" Nov 02 21:45:46.007008 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:46.006965 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:45:46.035281 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:46.035238 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:46.153152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:46.142678 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" event={"ID":"e9c858eb-093f-4e85-8338-094bb6743dfa","Type":"ContainerStarted","Data":"d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e"} Nov 02 21:45:46.153152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:46.142776 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:46.195539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:46.195512 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:46.255243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:46.255187 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" podStartSLOduration=3.255170922 podStartE2EDuration="3.255170922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:46.254563648 +0000 UTC m=+685.216113971" watchObservedRunningTime="2026-11-02 21:45:46.255170922 +0000 UTC m=+685.216721242" Nov 02 21:45:47.081069 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:47.081037 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 21:45:49.061807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:49.061770 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 21:45:49.275216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:49.275168 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:50.778531 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:50.778497 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:50.957657 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:50.957623 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:52.179733 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:52.179701 4285 generic.go:356] "Generic (PLEG): container finished" podID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerID="3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b" exitCode=0 Nov 02 21:45:52.180205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:52.179774 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" event={"ID":"700c6c8d-fb45-4029-9204-ad3e41f0657a","Type":"ContainerDied","Data":"3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b"} Nov 02 21:45:52.180205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:52.180163 4285 scope.go:122] "RemoveContainer" containerID="3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b" Nov 02 21:45:52.181437 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:52.181407 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:52.181564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:52.181493 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:52.181564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:52.181542 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:53.186730 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:53.186701 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" event={"ID":"700c6c8d-fb45-4029-9204-ad3e41f0657a","Type":"ContainerStarted","Data":"9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1"} Nov 02 21:45:53.237156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:53.236562 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:53.396945 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:53.396906 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:53.526151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:53.525659 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:53.535148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:53.527140 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:45:53.535148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:53.527197 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 21:45:54.265231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:54.265201 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:54.437991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:54.437955 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:54.524209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:54.523918 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:45:54.524209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:54.523981 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 21:45:54.596776 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:54.596739 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:54.982795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:54.982762 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:55.524899 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:55.524863 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:45:55.525663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:55.525632 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 21:45:56.153376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:56.150906 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 21:45:56.519434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:56.519400 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:45:56.519744 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:56.519714 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 21:45:57.531151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:57.526441 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:45:57.531151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:57.526504 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 02 21:45:58.156542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.156502 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 21:45:58.526879 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.526849 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:58.527819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.527797 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 21:45:58.931051 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.930975 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd"] Nov 02 21:45:58.934632 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.934616 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:58.989895 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.989860 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-64438f-serving-cert\"" Nov 02 21:45:58.993358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:58.993329 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd"] Nov 02 21:45:59.110379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.110344 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.110530 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.110385 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.110530 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.110508 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.110602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.110539 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frjd\" (UniqueName: \"kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.154969 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.154945 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:59.211297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.211265 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.211489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.211304 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.211489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.211347 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.211489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.211373 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frjd\" (UniqueName: \"kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.211655 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:59.211492 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-serving-cert: secret "maas-api-e2e-worker-w3-64438f-serving-cert" not found Nov 02 21:45:59.211655 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:45:59.211579 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls podName:b597d323-bab6-4636-8207-b182fbdec288 nodeName:}" failed. No retries permitted until 2026-11-02 21:45:59.711555213 +0000 UTC m=+698.673105525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls") pod "maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" (UID: "b597d323-bab6-4636-8207-b182fbdec288") : secret "maas-api-e2e-worker-w3-64438f-serving-cert" not found Nov 02 21:45:59.212150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.212114 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.213969 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.213948 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.268001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.267970 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frjd\" (UniqueName: \"kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.307754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.307725 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:59.486940 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.486860 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:59.716287 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.716247 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.718632 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.718609 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:45:59.847611 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:45:59.847520 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:46:00.226903 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:46:00.226873 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb597d323_bab6_4636_8207_b182fbdec288.slice/crio-d8c2733b5426567247af16479fda48df4428cb99eae701be2d81b6e09c36f43f WatchSource:0}: Error finding container d8c2733b5426567247af16479fda48df4428cb99eae701be2d81b6e09c36f43f: Status 404 returned error can't find the container with id d8c2733b5426567247af16479fda48df4428cb99eae701be2d81b6e09c36f43f Nov 02 21:46:00.315189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:00.315160 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd"] Nov 02 21:46:01.231944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:01.231915 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" event={"ID":"b597d323-bab6-4636-8207-b182fbdec288","Type":"ContainerStarted","Data":"7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4"} Nov 02 21:46:01.232335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:01.231962 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" event={"ID":"b597d323-bab6-4636-8207-b182fbdec288","Type":"ContainerStarted","Data":"d8c2733b5426567247af16479fda48df4428cb99eae701be2d81b6e09c36f43f"} Nov 02 21:46:01.232335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:01.232057 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:46:01.363233 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:01.363119 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" podStartSLOduration=2.569328521 podStartE2EDuration="3.36310461s" totalImagesPullingTime="793.776089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:01.361876163 +0000 UTC m=+700.323426486" watchObservedRunningTime="2026-11-02 21:46:01.36310461 +0000 UTC m=+700.324654930" Nov 02 21:46:03.948348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:03.948313 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz"] Nov 02 21:46:03.952167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:03.952149 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:03.965914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:03.965893 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:03.966035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:03.965936 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:03.966035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:03.965960 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bp2\" (UniqueName: \"kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:03.966190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:03.966099 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.056692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.056662 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-10fdd5-serving-cert\"" Nov 02 21:46:04.059168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.059144 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz"] Nov 02 21:46:04.067167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.067142 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.067288 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.067176 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bp2\" (UniqueName: \"kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.067288 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.067205 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.067288 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:46:04.067272 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-serving-cert: secret "maas-api-e2e-worker-w4-10fdd5-serving-cert" not found Nov 02 21:46:04.067457 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.067301 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.067457 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:46:04.067329 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls podName:ad87c76d-6364-4168-963b-8c9ba2255a5a nodeName:}" failed. No retries permitted until 2026-11-02 21:46:04.567315203 +0000 UTC m=+703.528865514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls") pod "maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" (UID: "ad87c76d-6364-4168-963b-8c9ba2255a5a") : secret "maas-api-e2e-worker-w4-10fdd5-serving-cert" not found Nov 02 21:46:04.067851 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.067833 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.069641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.069618 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.123945 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.123914 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bp2\" (UniqueName: \"kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.174452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.174425 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.359951 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.359873 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.552294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.552266 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.572815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.572789 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.575080 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.575055 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:04.864574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:04.864541 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:05.237271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:05.237248 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz"] Nov 02 21:46:05.239689 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:46:05.239663 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad87c76d_6364_4168_963b_8c9ba2255a5a.slice/crio-3dfdd94c796c74e6bd523809dd228b0fb42c18a45d11f7faf445ed798fb043cf WatchSource:0}: Error finding container 3dfdd94c796c74e6bd523809dd228b0fb42c18a45d11f7faf445ed798fb043cf: Status 404 returned error can't find the container with id 3dfdd94c796c74e6bd523809dd228b0fb42c18a45d11f7faf445ed798fb043cf Nov 02 21:46:05.254545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:05.254520 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" event={"ID":"ad87c76d-6364-4168-963b-8c9ba2255a5a","Type":"ContainerStarted","Data":"3dfdd94c796c74e6bd523809dd228b0fb42c18a45d11f7faf445ed798fb043cf"} Nov 02 21:46:06.144346 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:06.144318 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:06.261283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:06.261252 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" event={"ID":"ad87c76d-6364-4168-963b-8c9ba2255a5a","Type":"ContainerStarted","Data":"137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a"} Nov 02 21:46:06.261753 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:06.261380 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:06.377831 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:06.377732 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" podStartSLOduration=2.52809258 podStartE2EDuration="3.377712639s" totalImagesPullingTime="849.620059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:06.375434126 +0000 UTC m=+705.336984447" watchObservedRunningTime="2026-11-02 21:46:06.377712639 +0000 UTC m=+705.339262961" Nov 02 21:46:06.380050 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:06.380024 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:07.244268 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:07.244237 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 21:46:09.298856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:09.298814 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:12.277409 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:12.277374 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 21:46:13.994369 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:13.994332 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:14.163280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:14.163091 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:14.336735 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:14.336646 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:16.005750 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:46:16.005719 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:46:17.586748 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:17.586704 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:18.174917 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:18.174882 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:18.349982 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:18.349938 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:19.322414 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:19.322376 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:20.395084 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.394988 4285 generic.go:356] "Generic (PLEG): container finished" podID="acb34325-144e-449c-be36-7240ee0f4acd" containerID="82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54" exitCode=0 Nov 02 21:46:20.395084 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.395031 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" event={"ID":"acb34325-144e-449c-be36-7240ee0f4acd","Type":"ContainerDied","Data":"82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54"} Nov 02 21:46:20.395651 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.395525 4285 scope.go:122] "RemoveContainer" containerID="82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54" Nov 02 21:46:20.396893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.396873 4285 generic.go:356] "Generic (PLEG): container finished" podID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerID="f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0" exitCode=0 Nov 02 21:46:20.396997 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.396942 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" event={"ID":"08ab0a7c-02dc-4187-8d69-5f4fe48cb164","Type":"ContainerDied","Data":"f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0"} Nov 02 21:46:20.397046 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.397009 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.397093 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.397079 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.397152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.397114 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.397317 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.397303 4285 scope.go:122] "RemoveContainer" containerID="f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0" Nov 02 21:46:20.398531 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.398507 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.398616 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.398565 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.398616 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:20.398595 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:21.277025 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.273536 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:21.404241 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.404204 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" event={"ID":"acb34325-144e-449c-be36-7240ee0f4acd","Type":"ContainerStarted","Data":"5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f"} Nov 02 21:46:21.406362 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.406339 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" event={"ID":"08ab0a7c-02dc-4187-8d69-5f4fe48cb164","Type":"ContainerStarted","Data":"efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd"} Nov 02 21:46:21.784820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.784782 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:46:21.786418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.786386 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:46:21.786509 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.786482 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 21:46:21.848155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.846641 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:46:21.848155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.846768 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:46:21.848155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:21.846808 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 21:46:22.784841 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:22.784806 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:46:22.785256 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:22.784872 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 21:46:22.848148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:22.847550 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:46:22.848148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:22.847623 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 21:46:23.582065 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:23.582021 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:23.786185 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:23.785691 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:46:23.786185 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:23.785768 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 21:46:23.853210 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:23.847284 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:46:23.853210 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:23.847355 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 21:46:24.439696 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:24.439650 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:24.785371 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:24.785324 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:46:24.785550 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:24.785385 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 21:46:24.846504 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:24.846462 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:46:24.847095 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:24.847064 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 21:46:24.983708 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:24.983659 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:46:25.784715 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:25.784681 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:46:25.784971 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:25.784738 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 21:46:25.845566 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:25.845514 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:46:25.845854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:25.845806 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 02 21:46:25.988792 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:25.988760 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:46:26.788946 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:26.788910 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:46:26.789937 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:26.789914 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 21:46:26.848742 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:26.848722 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:46:26.849540 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:26.849522 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 21:46:28.009971 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:28.009932 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:28.580615 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:28.580583 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:28.744501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:28.744464 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:28.913098 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:28.913016 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:29.640294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:29.640261 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:33.329155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:33.328243 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:33.843932 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:33.843894 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:34.008251 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:34.008219 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:34.909912 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:34.909868 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:39.571863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.571836 4285 generic.go:356] "Generic (PLEG): container finished" podID="9e29ece0-7361-4bae-8651-18356ed45965" containerID="2740ef9f1a718fc9c17d7684fcfedc15c552b8c307d5bc9510bc4008eb18f940" exitCode=0 Nov 02 21:46:39.572216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.571906 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" event={"ID":"9e29ece0-7361-4bae-8651-18356ed45965","Type":"ContainerDied","Data":"2740ef9f1a718fc9c17d7684fcfedc15c552b8c307d5bc9510bc4008eb18f940"} Nov 02 21:46:39.572341 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.572323 4285 scope.go:122] "RemoveContainer" containerID="2740ef9f1a718fc9c17d7684fcfedc15c552b8c307d5bc9510bc4008eb18f940" Nov 02 21:46:39.573599 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.573574 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:39.573662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.573639 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:39.573700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.573671 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:39.935427 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:39.935347 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:40.579674 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:40.579641 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" event={"ID":"9e29ece0-7361-4bae-8651-18356ed45965","Type":"ContainerStarted","Data":"e5330c2c0bc5386983a022108b2a33dc7397474c2e3445e6dd68418a2a808a3b"} Nov 02 21:46:40.989654 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:40.989614 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:46:41.104527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:41.104489 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:46:41.105641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:41.105616 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:46:41.105703 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:41.105679 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 21:46:42.104454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:42.104418 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:46:42.104841 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:42.104472 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 21:46:42.548083 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:42.548049 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:43.104035 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:43.103998 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:46:43.104237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:43.104065 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 21:46:43.142826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:43.142788 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:43.301803 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:43.301754 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:44.105851 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:44.105812 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:46:44.106026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:44.105873 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 21:46:45.097607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:45.097567 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:45.103929 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:45.103904 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:46:45.104113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:45.104094 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 21:46:46.005396 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:46:46.005363 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:46:46.108034 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:46.108000 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:46:46.109086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:46.109068 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 21:46:47.050508 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:47.050474 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:48.453724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:48.453686 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:49.069261 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:49.069222 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:49.238765 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:49.238713 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:50.175313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:50.175271 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:54.870908 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:54.870875 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:55.200625 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:55.200593 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:55.409157 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:55.409098 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:55.576858 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:55.576763 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:59.917523 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:46:59.917487 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:04.907586 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:04.907515 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:05.073303 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:05.073269 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:05.665618 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:05.665589 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:05.894154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:05.894097 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:06.057796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:06.057760 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:06.479311 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:06.479277 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:06.751767 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:06.751684 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:09.699537 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:09.699502 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:14.378525 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:14.378484 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:16.005156 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:47:16.005107 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:47:19.014449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:19.014410 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:20.117167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.117113 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf"] Nov 02 21:47:20.119647 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.119631 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.183650 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.183614 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"openshift-service-ca.crt\"" Nov 02 21:47:20.198318 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.198284 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.198478 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.198401 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.198548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.198491 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.198619 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.198553 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.198697 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.198682 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.198763 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.198715 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsb4\" (UniqueName: \"kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.231450 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.231414 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"kube-root-ca.crt\"" Nov 02 21:47:20.231599 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.231504 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"default-dockercfg-h76tv\"" Nov 02 21:47:20.231599 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.231574 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb8112\"/\"premium-w6-fb8112-kserve-self-signed-certs\"" Nov 02 21:47:20.235663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.235637 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf"] Nov 02 21:47:20.302342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.302271 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.302785 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.302711 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.302890 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.302801 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.302890 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.302857 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.302983 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.302919 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.303081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.303064 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.303184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.303096 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsb4\" (UniqueName: \"kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.303860 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.303833 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.303860 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.303850 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.305354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.305331 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:20.305813 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.305792 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.306345 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.306325 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.360664 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.360635 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsb4\" (UniqueName: \"kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4\") pod \"premium-w6-fb8112-kserve-697f6d85d8-mp7xf\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.439594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.439513 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:20.840184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:20.840160 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf"] Nov 02 21:47:20.841501 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:47:20.841478 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod142239e4_164d_4f9c_a2e0_f4e1537b4796.slice/crio-040453f5c7ad0e7615dd3658b0d05ea48245ee98bef517178d6f46989a2b04f2 WatchSource:0}: Error finding container 040453f5c7ad0e7615dd3658b0d05ea48245ee98bef517178d6f46989a2b04f2: Status 404 returned error can't find the container with id 040453f5c7ad0e7615dd3658b0d05ea48245ee98bef517178d6f46989a2b04f2 Nov 02 21:47:21.823980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:21.823943 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" event={"ID":"142239e4-164d-4f9c-a2e0-f4e1537b4796","Type":"ContainerStarted","Data":"2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4"} Nov 02 21:47:21.824376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:21.823990 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" event={"ID":"142239e4-164d-4f9c-a2e0-f4e1537b4796","Type":"ContainerStarted","Data":"040453f5c7ad0e7615dd3658b0d05ea48245ee98bef517178d6f46989a2b04f2"} Nov 02 21:47:21.937543 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:21.937482 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podStartSLOduration=1.937466568 podStartE2EDuration="1.937466568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:21.935459939 +0000 UTC m=+780.897010259" watchObservedRunningTime="2026-11-02 21:47:21.937466568 +0000 UTC m=+780.899016890" Nov 02 21:47:22.439633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:22.439598 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:23.161486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:23.161452 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:23.714142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:23.714101 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:28.451475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:28.424462 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:30.440646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:30.440615 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:30.457682 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:30.457649 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:30.884179 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:30.884148 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:47:33.023299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.023270 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:33.314021 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.313991 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7"] Nov 02 21:47:33.316924 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.316907 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.372017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.371980 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"openshift-service-ca.crt\"" Nov 02 21:47:33.421593 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.421566 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"simulator-w4-10fdd5-kserve-self-signed-certs\"" Nov 02 21:47:33.421749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.421606 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"default-dockercfg-vs5jj\"" Nov 02 21:47:33.421749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.421655 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"kube-root-ca.crt\"" Nov 02 21:47:33.424166 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.424142 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7"] Nov 02 21:47:33.427309 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.427282 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:33.462783 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.462755 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.462894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.462800 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.462965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.462907 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.462965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.462940 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.462965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.462957 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdtg\" (UniqueName: \"kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.463145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.463134 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.563927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.563889 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.563927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.563927 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.563944 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdtg\" (UniqueName: \"kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.564018 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.564037 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.564080 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.564389 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564531 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.564510 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.564609 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.564596 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.566720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.566687 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.566720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.566709 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.620219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.620194 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdtg\" (UniqueName: \"kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg\") pod \"simulator-w4-10fdd5-kserve-cf85d484c-mb2p7\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.628192 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.628160 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:33.998158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:33.998115 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7"] Nov 02 21:47:33.999767 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:47:33.999743 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6784d5f2_659f_45cf_b603_99f19eae6473.slice/crio-bf6c9f502aa104b17bc8aa679f3be1013ec77ec41ff53d50b9b677e26b7166aa WatchSource:0}: Error finding container bf6c9f502aa104b17bc8aa679f3be1013ec77ec41ff53d50b9b677e26b7166aa: Status 404 returned error can't find the container with id bf6c9f502aa104b17bc8aa679f3be1013ec77ec41ff53d50b9b677e26b7166aa Nov 02 21:47:34.910379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:34.910337 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" event={"ID":"6784d5f2-659f-45cf-b603-99f19eae6473","Type":"ContainerStarted","Data":"4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463"} Nov 02 21:47:34.910767 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:34.910389 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" event={"ID":"6784d5f2-659f-45cf-b603-99f19eae6473","Type":"ContainerStarted","Data":"bf6c9f502aa104b17bc8aa679f3be1013ec77ec41ff53d50b9b677e26b7166aa"} Nov 02 21:47:35.023913 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:35.023860 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podStartSLOduration=2.023845419 podStartE2EDuration="2.023845419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:35.021937338 +0000 UTC m=+793.983487659" watchObservedRunningTime="2026-11-02 21:47:35.023845419 +0000 UTC m=+793.985395738" Nov 02 21:47:35.628540 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:35.628503 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:37.695564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:37.695537 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:40.082718 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:40.082685 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:42.320715 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:42.320670 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:43.628868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:43.628834 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:43.642227 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:43.642190 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:43.970282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:43.970257 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:47:46.011444 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:47:46.011406 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:47:46.990735 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:46.990695 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:51.701685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:51.701646 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:56.336511 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:56.336478 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:57.592890 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.592856 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz"] Nov 02 21:47:57.595644 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.595627 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.698545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.698512 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"default-dockercfg-2xl7c\"" Nov 02 21:47:57.698545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.698512 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"kube-root-ca.crt\"" Nov 02 21:47:57.698745 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.698586 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"premium-w2-9cc56b-kserve-self-signed-certs\"" Nov 02 21:47:57.698745 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.698598 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9cc56b\"/\"openshift-service-ca.crt\"" Nov 02 21:47:57.701022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.701002 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz"] Nov 02 21:47:57.709732 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.709709 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:57.733102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.733075 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4zk\" (UniqueName: \"kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.733244 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.733119 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.733292 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.733267 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.733334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.733295 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.733410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.733397 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.733502 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.733485 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.834391 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.834351 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.834563 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.834435 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.834563 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.834490 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4zk\" (UniqueName: \"kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.834563 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.834520 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.834739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.834605 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.834739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.834636 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.835033 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.835005 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.835108 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.835053 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.835246 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.835216 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.836706 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.836683 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.837573 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.837550 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.890867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.890800 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4zk\" (UniqueName: \"kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk\") pod \"premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:57.906843 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:57.906816 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:47:58.315301 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:58.315274 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz"] Nov 02 21:47:58.317337 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:47:58.317310 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9521ca7_741b_4807_bdde_0aa350f8f3a8.slice/crio-075acbc68210a3e0db0bfd718530859afa2ef4683930611c7126399a641dc68d WatchSource:0}: Error finding container 075acbc68210a3e0db0bfd718530859afa2ef4683930611c7126399a641dc68d: Status 404 returned error can't find the container with id 075acbc68210a3e0db0bfd718530859afa2ef4683930611c7126399a641dc68d Nov 02 21:47:59.049302 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:59.049260 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" event={"ID":"c9521ca7-741b-4807-bdde-0aa350f8f3a8","Type":"ContainerStarted","Data":"41c48f35317758685117b7b0ef58b3039a7f891bdf7dae6a41ef0c17b89e1093"} Nov 02 21:47:59.049707 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:59.049311 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" event={"ID":"c9521ca7-741b-4807-bdde-0aa350f8f3a8","Type":"ContainerStarted","Data":"075acbc68210a3e0db0bfd718530859afa2ef4683930611c7126399a641dc68d"} Nov 02 21:47:59.163743 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:59.163693 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podStartSLOduration=2.163680175 podStartE2EDuration="2.163680175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:59.160922429 +0000 UTC m=+818.122472749" watchObservedRunningTime="2026-11-02 21:47:59.163680175 +0000 UTC m=+818.125230532" Nov 02 21:47:59.906924 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:47:59.906886 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:48:01.015672 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:01.015638 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:05.690465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:05.690428 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:06.813698 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:06.813662 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:07.907716 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:07.907684 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:48:07.919987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:07.919967 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:48:08.121615 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:08.121585 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:48:10.422765 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:10.422713 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:14.817266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:14.817209 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:15.100722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:15.100625 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:16.004834 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:48:16.004802 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:48:19.797436 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:19.797390 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:24.531997 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:24.531938 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:29.217808 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:29.217774 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:31.603354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:31.603322 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:33.884187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:33.884091 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:38.517683 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:38.517656 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:43.047805 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:43.047770 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:43.223316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:48:43.223280 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:46.005384 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:48:46.005348 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:49:00.425215 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:00.425172 4285 generic.go:356] "Generic (PLEG): container finished" podID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerID="7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4" exitCode=0 Nov 02 21:49:00.425777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:00.425252 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" event={"ID":"1b5549b0-1277-4bbe-9600-2a9eb00575a0","Type":"ContainerDied","Data":"7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4"} Nov 02 21:49:00.425777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:00.425665 4285 scope.go:122] "RemoveContainer" containerID="7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4" Nov 02 21:49:01.432708 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:01.432671 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" event={"ID":"1b5549b0-1277-4bbe-9600-2a9eb00575a0","Type":"ContainerStarted","Data":"c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2"} Nov 02 21:49:01.433101 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:01.432884 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:49:05.280248 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.280213 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk"] Nov 02 21:49:05.284195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.284177 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.356687 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.356653 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.356852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.356712 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.356907 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.356865 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.356955 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.356924 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.357088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.357059 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8zd\" (UniqueName: \"kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.357224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.357204 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.387365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.387332 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"premium-w4-10fdd5-kserve-self-signed-certs\"" Nov 02 21:49:05.389936 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.389905 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk"] Nov 02 21:49:05.393702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.393679 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:05.458240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458208 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458287 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458323 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458352 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8zd\" (UniqueName: \"kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458405 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458421 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458771 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458751 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458857 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458789 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.458857 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.458807 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.460685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.460663 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.460987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.460965 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.514944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.514905 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8zd\" (UniqueName: \"kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd\") pod \"premium-w4-10fdd5-kserve-84559c6446-g88lk\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:05.596774 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:05.596690 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:06.024793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:06.024766 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk"] Nov 02 21:49:06.026539 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:49:06.026507 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfceaaaee_d6ec_41a0_92cd_63dad84030af.slice/crio-fada2780ccb4b0569baeb952d335058bd06fe8a3cad046ad26fb508042c91700 WatchSource:0}: Error finding container fada2780ccb4b0569baeb952d335058bd06fe8a3cad046ad26fb508042c91700: Status 404 returned error can't find the container with id fada2780ccb4b0569baeb952d335058bd06fe8a3cad046ad26fb508042c91700 Nov 02 21:49:06.463269 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:06.463226 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" event={"ID":"fceaaaee-d6ec-41a0-92cd-63dad84030af","Type":"ContainerStarted","Data":"660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260"} Nov 02 21:49:06.463679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:06.463281 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" event={"ID":"fceaaaee-d6ec-41a0-92cd-63dad84030af","Type":"ContainerStarted","Data":"fada2780ccb4b0569baeb952d335058bd06fe8a3cad046ad26fb508042c91700"} Nov 02 21:49:06.579164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:06.579096 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podStartSLOduration=1.5790833279999998 podStartE2EDuration="1.579083328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:06.577658794 +0000 UTC m=+885.539209115" watchObservedRunningTime="2026-11-02 21:49:06.579083328 +0000 UTC m=+885.540633647" Nov 02 21:49:06.597829 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:06.597798 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:07.159746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:07.159713 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:07.385602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:07.385569 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:12.439905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:12.439875 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 21:49:15.597852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:15.597817 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:15.610531 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:15.610508 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:16.006864 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:49:16.006829 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:49:16.529386 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:16.529351 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:49:17.044299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:17.044266 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:21.689666 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:21.689622 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:26.499022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:26.498986 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.239547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:31.239500 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:35.892455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:35.892420 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:40.622529 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:40.622495 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:45.400042 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:45.400004 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:46.006033 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:49:46.005998 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:49:50.203359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:50.203325 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:54.896352 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:54.896312 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:55.287772 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.287740 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8"] Nov 02 21:49:55.291739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.291720 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.343602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.343576 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f48c4d-openshift-default-dockercfg-xpv97\"" Nov 02 21:49:55.346985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.346952 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347029 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79kd\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347055 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347070 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347214 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347269 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347252 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347327 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347306 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347363 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347336 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.347420 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.347407 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.396506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.396476 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8"] Nov 02 21:49:55.449300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449259 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449316 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449342 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449356 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449389 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449469 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449550 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79kd\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449589 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449615 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.449836 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.449793 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.450099 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.450077 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.450195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.450156 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.450241 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.450214 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.450348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.450326 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.451637 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.451614 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.451919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.451902 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.505404 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.505381 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.505525 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.505509 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79kd\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:55.604212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:55.604153 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:56.025077 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:56.025050 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8"] Nov 02 21:49:56.026558 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:49:56.026527 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc24004_27d8_41b4_bcca_6270f816e928.slice/crio-8785c4cfdbc08b9e8c6ed997631ac7b37335a550507b5e4b18a70509c7975d74 WatchSource:0}: Error finding container 8785c4cfdbc08b9e8c6ed997631ac7b37335a550507b5e4b18a70509c7975d74: Status 404 returned error can't find the container with id 8785c4cfdbc08b9e8c6ed997631ac7b37335a550507b5e4b18a70509c7975d74 Nov 02 21:49:56.028860 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:56.028832 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:56.028940 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:56.028891 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:56.028940 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:56.028921 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:56.811722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:56.811683 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" event={"ID":"bfc24004-27d8-41b4-bcca-6270f816e928","Type":"ContainerStarted","Data":"0ba551c597355f231d7b647970cb1eae10117207e5f287812164294d0fd07829"} Nov 02 21:49:56.811935 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:56.811733 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" event={"ID":"bfc24004-27d8-41b4-bcca-6270f816e928","Type":"ContainerStarted","Data":"8785c4cfdbc08b9e8c6ed997631ac7b37335a550507b5e4b18a70509c7975d74"} Nov 02 21:49:57.001962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:57.001902 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podStartSLOduration=2.001887197 podStartE2EDuration="2.001887197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:57.00047276 +0000 UTC m=+935.962023101" watchObservedRunningTime="2026-11-02 21:49:57.001887197 +0000 UTC m=+935.963437517" Nov 02 21:49:57.604729 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:57.604688 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:49:57.606373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:57.606343 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:49:57.606488 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:57.606407 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 21:49:58.605258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:58.605222 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:49:58.605626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:58.605275 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 21:49:59.592811 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:59.592779 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:59.604505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:59.604471 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:49:59.604658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:49:59.604527 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 21:50:00.604815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:00.604777 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:50:00.605220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:00.604834 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 21:50:01.604595 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:01.604560 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:50:01.604839 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:01.604617 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 21:50:02.609066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:02.609037 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:50:02.609573 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:02.609543 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:50:02.610319 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:02.610297 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 21:50:04.314948 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:04.314861 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:07.544456 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:07.544424 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:08.967594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:08.967564 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:13.698750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:13.698719 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:16.005892 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:50:16.005859 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:50:18.417972 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:18.417941 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:23.118380 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:23.118339 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:27.825448 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:27.825398 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:32.512197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:32.512163 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:32.668257 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:32.668222 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:33.128076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.128042 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd"] Nov 02 21:50:33.133064 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.133044 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.190957 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.190881 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd"] Nov 02 21:50:33.196063 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.196033 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbsx\" (UniqueName: \"kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.196235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.196088 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.196303 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.196230 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.236865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.236839 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:50:33.237010 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.236878 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:50:33.296807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.296780 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbsx\" (UniqueName: \"kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.296942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.296819 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.296942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.296857 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.297653 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.297625 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.299175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.299149 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.352788 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.352756 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbsx\" (UniqueName: \"kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx\") pod \"maas-api-revoke-sub-546d97f9-g6qfd\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.450139 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.450107 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:33.830831 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:33.830733 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd"] Nov 02 21:50:33.843281 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:50:33.843248 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5642d0_5774_4422_825b_a7395048ab67.slice/crio-7d472388a054aa6f6cd770da84ea1da33f23b46dc28702fc463f778740874b2e WatchSource:0}: Error finding container 7d472388a054aa6f6cd770da84ea1da33f23b46dc28702fc463f778740874b2e: Status 404 returned error can't find the container with id 7d472388a054aa6f6cd770da84ea1da33f23b46dc28702fc463f778740874b2e Nov 02 21:50:34.071009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:34.070971 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" event={"ID":"3b5642d0-5774-4422-825b-a7395048ab67","Type":"ContainerStarted","Data":"7d472388a054aa6f6cd770da84ea1da33f23b46dc28702fc463f778740874b2e"} Nov 02 21:50:35.081681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:35.081646 4285 generic.go:356] "Generic (PLEG): container finished" podID="3b5642d0-5774-4422-825b-a7395048ab67" containerID="a68d12c3f8926255204db2bcc7043001865b3db33c416a3e750bd56c550a4729" exitCode=0 Nov 02 21:50:35.082066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:35.081739 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" event={"ID":"3b5642d0-5774-4422-825b-a7395048ab67","Type":"ContainerDied","Data":"a68d12c3f8926255204db2bcc7043001865b3db33c416a3e750bd56c550a4729"} Nov 02 21:50:36.579837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.579521 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:36.621473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.620679 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx\" (UniqueName: \"kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx\") pod \"3b5642d0-5774-4422-825b-a7395048ab67\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " Nov 02 21:50:36.621473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.620762 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca\") pod \"3b5642d0-5774-4422-825b-a7395048ab67\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " Nov 02 21:50:36.621473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.620849 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca\") pod \"3b5642d0-5774-4422-825b-a7395048ab67\" (UID: \"3b5642d0-5774-4422-825b-a7395048ab67\") " Nov 02 21:50:36.622268 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.621930 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca" pod "3b5642d0-5774-4422-825b-a7395048ab67" (UID: "3b5642d0-5774-4422-825b-a7395048ab67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:36.658068 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.652333 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx" pod "3b5642d0-5774-4422-825b-a7395048ab67" (UID: "3b5642d0-5774-4422-825b-a7395048ab67"). InnerVolumeSpecName "kube-api-access-mwbsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:36.682147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.681274 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca" pod "3b5642d0-5774-4422-825b-a7395048ab67" (UID: "3b5642d0-5774-4422-825b-a7395048ab67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:36.733950 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.733864 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbsx\" (UniqueName: \"kubernetes.io/projected/3b5642d0-5774-4422-825b-a7395048ab67-kube-api-access-mwbsx\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:36.733950 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.733906 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:36.733950 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:36.733922 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5642d0-5774-4422-825b-a7395048ab67-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:50:37.105562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:37.105419 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" Nov 02 21:50:37.105562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:37.105429 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd" event={"ID":"3b5642d0-5774-4422-825b-a7395048ab67","Type":"ContainerDied","Data":"7d472388a054aa6f6cd770da84ea1da33f23b46dc28702fc463f778740874b2e"} Nov 02 21:50:37.105562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:37.105500 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d472388a054aa6f6cd770da84ea1da33f23b46dc28702fc463f778740874b2e" Nov 02 21:50:37.303741 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:37.303695 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:41.994415 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:41.994379 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:46.005191 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:50:46.005156 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:50:46.704946 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:46.704907 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:50.690397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:50.690365 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:51.414963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:51.414922 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:53.740146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.733709 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk"] Nov 02 21:50:53.757149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.756849 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3b5642d0-5774-4422-825b-a7395048ab67" containerName="revoke-keys" Nov 02 21:50:53.757149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.756878 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5642d0-5774-4422-825b-a7395048ab67" containerName="revoke-keys" Nov 02 21:50:53.757149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.757067 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5642d0-5774-4422-825b-a7395048ab67" containerName="revoke-keys" Nov 02 21:50:53.761020 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.760995 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:53.803814 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.803781 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk"] Nov 02 21:50:53.882485 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.882453 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"distinct-w4-10fdd5-kserve-self-signed-certs\"" Nov 02 21:50:53.883445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.883416 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:53.935031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.907816 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:53.935510 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.935478 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:53.935771 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.935741 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:53.935961 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.935943 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:53.936165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.936138 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthqk\" (UniqueName: \"kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:53.936437 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:53.936416 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.040177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.040048 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.040485 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.040462 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.040777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.040758 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.040919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.040904 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.041096 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.041067 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.041272 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.041253 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthqk\" (UniqueName: \"kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.044525 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.044497 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.045170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.045084 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.045325 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.045146 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.058912 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.058891 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.066214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.066184 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.136790 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.136759 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthqk\" (UniqueName: \"kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk\") pod \"distinct-w4-10fdd5-kserve-754d654d89-mfzgk\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.434220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.432398 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:54.894318 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:50:54.894280 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eb17e5a_fbf5_4fc0_b8e7_e51a0ae3be7b.slice/crio-deece6b6c5c7281b1cf6fd59d3e736664e4a37e5ded2327cf2c19df397be872f WatchSource:0}: Error finding container deece6b6c5c7281b1cf6fd59d3e736664e4a37e5ded2327cf2c19df397be872f: Status 404 returned error can't find the container with id deece6b6c5c7281b1cf6fd59d3e736664e4a37e5ded2327cf2c19df397be872f Nov 02 21:50:54.896195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:54.896172 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk"] Nov 02 21:50:55.237643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:55.237604 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" event={"ID":"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b","Type":"ContainerStarted","Data":"af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f"} Nov 02 21:50:55.237900 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:55.237880 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" event={"ID":"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b","Type":"ContainerStarted","Data":"deece6b6c5c7281b1cf6fd59d3e736664e4a37e5ded2327cf2c19df397be872f"} Nov 02 21:50:55.358691 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:55.358629 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podStartSLOduration=2.358598331 podStartE2EDuration="2.358598331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:55.358352262 +0000 UTC m=+994.319902584" watchObservedRunningTime="2026-11-02 21:50:55.358598331 +0000 UTC m=+994.320148653" Nov 02 21:50:55.432969 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:55.432934 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:50:56.770241 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:50:56.770203 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:01.800493 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:01.800449 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:04.432680 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:04.432648 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:51:04.445826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:04.445800 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:51:05.317479 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:05.317449 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:51:06.498240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:06.498203 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:07.884243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:07.884211 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:10.513694 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.513661 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6"] Nov 02 21:51:10.518418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.518391 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543199 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543167 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543215 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543330 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543409 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543498 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543476 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcl25\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543683 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543663 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543748 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543721 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543821 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.543896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.543861 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.622539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.622505 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9cc56b-openshift-default-dockercfg-hl779\"" Nov 02 21:51:10.625188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.625163 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6"] Nov 02 21:51:10.645136 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645087 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645244 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645184 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcl25\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645244 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645218 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645269 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645335 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645459 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645373 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645510 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645489 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645527 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645577 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645631 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645708 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.645898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.645806 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.646207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.646186 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.646327 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.646272 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.648399 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.648380 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.648720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.648701 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.701876 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.701852 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.702012 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.701910 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcl25\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:10.835957 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:10.835866 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:11.301032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:11.300997 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6"] Nov 02 21:51:11.306429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:11.306382 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.306686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:11.306663 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.306945 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:11.306915 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:11.378371 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:11.378337 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" event={"ID":"073a27af-aa00-4982-91b0-9509297881f6","Type":"ContainerStarted","Data":"2acaf3982573e11e0e3c68668b6fd81e7e63b97b8112ec322055f6d0f74760be"} Nov 02 21:51:11.394521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:11.394489 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:12.453150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:12.450343 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" event={"ID":"073a27af-aa00-4982-91b0-9509297881f6","Type":"ContainerStarted","Data":"aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8"} Nov 02 21:51:12.643283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:12.643217 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podStartSLOduration=2.64318724 podStartE2EDuration="2.64318724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:12.635752278 +0000 UTC m=+1011.597302598" watchObservedRunningTime="2026-11-02 21:51:12.64318724 +0000 UTC m=+1011.604737576" Nov 02 21:51:12.836417 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:12.836322 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:12.838740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:12.838709 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 21:51:12.838965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:12.838936 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 21:51:13.838496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:13.838455 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 21:51:13.839001 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:13.838512 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 21:51:14.837623 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:14.837582 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 21:51:14.837844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:14.837685 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 21:51:15.839148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:15.838552 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 21:51:15.839148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:15.838615 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 21:51:16.005834 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:51:16.005796 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:51:16.386237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:16.386204 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:16.837516 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:16.837476 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 21:51:16.837695 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:16.837531 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 21:51:17.837684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:17.837646 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 21:51:17.838151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:17.837721 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 21:51:18.840878 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:18.840847 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:18.841303 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:18.841229 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:18.841888 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:18.841872 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 21:51:21.093005 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:21.092965 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:25.765169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:25.765107 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:26.311077 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.311041 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4"] Nov 02 21:51:26.316067 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.316046 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.402715 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.402683 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403140 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403106 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403227 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdt5m\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403336 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403318 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403381 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403459 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403529 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403497 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403585 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403526 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.403760 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.403720 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.406495 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.406462 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4"] Nov 02 21:51:26.504486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504459 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504519 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504571 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdt5m\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504625 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504656 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504740 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504780 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.504815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504809 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.505032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504851 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.505032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.504885 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.505409 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.505380 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.505542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.505405 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.505542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.505465 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.505678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.505662 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.507281 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.507261 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.507671 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.507650 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.563759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.563694 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdt5m\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.564055 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.564037 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:26.629305 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:26.629271 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:27.164149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.163683 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4"] Nov 02 21:51:27.178379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.177443 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:27.178379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.177539 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:27.178379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.177597 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:27.569288 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.569246 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" event={"ID":"12b7c971-1048-4f55-9bd3-d8f705f26f80","Type":"ContainerStarted","Data":"ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224"} Nov 02 21:51:27.569640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.569614 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" event={"ID":"12b7c971-1048-4f55-9bd3-d8f705f26f80","Type":"ContainerStarted","Data":"5eaaed47f72b5fd5a5dd324c61b418a93c222c77e12a9ce57e6cd74357e742c6"} Nov 02 21:51:27.630348 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:27.630312 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:28.638303 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:28.638252 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:51:28.638835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:28.638339 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 21:51:29.632484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:29.632449 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:51:29.633451 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:29.633416 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 21:51:30.631539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:30.631501 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:51:30.632148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:30.632091 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 21:51:30.744707 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:30.744671 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:31.637156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:31.636521 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:51:31.637156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:31.636598 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 21:51:32.633496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:32.633458 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:51:32.633769 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:32.633736 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 21:51:33.629625 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:33.629537 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:51:33.629625 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:33.629607 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 21:51:34.634358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:34.634324 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:34.634770 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:34.634698 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:34.635400 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:34.635382 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 21:51:34.851513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:34.851453 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podStartSLOduration=8.851439106 podStartE2EDuration="8.851439106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:27.689403074 +0000 UTC m=+1026.650953400" watchObservedRunningTime="2026-11-02 21:51:34.851439106 +0000 UTC m=+1033.812989425" Nov 02 21:51:35.421072 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:35.421039 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:40.103187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:40.103153 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:41.090480 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.090444 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl"] Nov 02 21:51:41.094293 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.094278 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.151796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.151769 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.151798 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.151828 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.151994 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.152040 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.152104 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.152172 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.152188 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.152295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.152283 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvqs\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.201872 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.201835 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl"] Nov 02 21:51:41.253648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253620 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.253819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253688 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.253819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253743 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.253819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253767 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.253819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253811 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvqs\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253876 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253901 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.253936 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254227 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.254139 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.254418 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.254442 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.254461 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.254488 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.254731 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.254688 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.256244 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.256223 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.257374 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.257354 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.309662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.309637 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.310100 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.310081 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvqs\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.408012 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.407934 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:41.830873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.830838 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl"] Nov 02 21:51:41.871481 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:51:41.871452 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbdd06f_423a_46b9_9ee2_1e07ec32dd0a.slice/crio-2744b57ea057948eda5e9729feb0556d6c655ae8f144fca1994b8fe0960a8928 WatchSource:0}: Error finding container 2744b57ea057948eda5e9729feb0556d6c655ae8f144fca1994b8fe0960a8928: Status 404 returned error can't find the container with id 2744b57ea057948eda5e9729feb0556d6c655ae8f144fca1994b8fe0960a8928 Nov 02 21:51:41.873955 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.873920 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:41.874037 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.873988 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:41.874037 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:41.874019 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:42.671444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:42.671404 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" event={"ID":"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a","Type":"ContainerStarted","Data":"d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf"} Nov 02 21:51:42.671444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:42.671468 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" event={"ID":"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a","Type":"ContainerStarted","Data":"2744b57ea057948eda5e9729feb0556d6c655ae8f144fca1994b8fe0960a8928"} Nov 02 21:51:42.892152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:42.892085 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podStartSLOduration=1.8920715160000001 podStartE2EDuration="1.892071516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:42.841570257 +0000 UTC m=+1041.803120614" watchObservedRunningTime="2026-11-02 21:51:42.892071516 +0000 UTC m=+1041.853621815" Nov 02 21:51:43.408756 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:43.408725 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:43.410389 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:43.410363 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:51:43.410499 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:43.410412 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 21:51:44.044471 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:44.044242 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:44.409153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:44.409046 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:51:44.409315 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:44.409104 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 21:51:44.837144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:44.836947 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:45.408287 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:45.408255 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:51:45.408755 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:45.408312 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 21:51:46.008384 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:51:46.008346 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:51:46.408578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:46.408502 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:51:46.408578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:46.408568 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 21:51:47.415147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:47.410585 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:51:47.415147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:47.410676 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 21:51:48.408818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:48.408780 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:51:48.409005 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:48.408847 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 21:51:49.750672 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:49.750630 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:50.004396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.004308 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:50.006252 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.006224 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:50.008892 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.008874 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 21:51:50.872849 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.871825 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf"] Nov 02 21:51:50.877520 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.877495 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.916639 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.916588 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf"] Nov 02 21:51:50.972152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.970461 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:50.973574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.973549 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"default-dockercfg-ls6zb\"" Nov 02 21:51:50.973907 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.973838 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"distinct-w3-64438f-kserve-self-signed-certs\"" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.978826 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.978916 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.978989 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.979045 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.979081 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wgd\" (UniqueName: \"kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.979213 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.979483 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"openshift-service-ca.crt\"" Nov 02 21:51:50.979597 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:50.979566 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-64438f\"/\"kube-root-ca.crt\"" Nov 02 21:51:51.080319 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.080281 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.080598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.080582 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.080732 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.080718 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.080832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.080819 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wgd\" (UniqueName: \"kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.080983 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.080971 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.081170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.081153 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.081702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.081678 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.082052 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.082033 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.083003 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.082745 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.085269 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.085248 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.085731 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.085710 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.139767 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.139698 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wgd\" (UniqueName: \"kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd\") pod \"distinct-w3-64438f-kserve-5c6bcf5955-dfnhf\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.195345 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.195314 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:51.695109 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.695076 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf"] Nov 02 21:51:51.697992 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:51:51.697955 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56500d5e_8f5f_4012_8077_e98730a4ab62.slice/crio-7e286a4276d0c8c4320373588914f4b8ae31d3d3418fdafe11aaca68467340c5 WatchSource:0}: Error finding container 7e286a4276d0c8c4320373588914f4b8ae31d3d3418fdafe11aaca68467340c5: Status 404 returned error can't find the container with id 7e286a4276d0c8c4320373588914f4b8ae31d3d3418fdafe11aaca68467340c5 Nov 02 21:51:51.781820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:51.781786 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" event={"ID":"56500d5e-8f5f-4012-8077-e98730a4ab62","Type":"ContainerStarted","Data":"7e286a4276d0c8c4320373588914f4b8ae31d3d3418fdafe11aaca68467340c5"} Nov 02 21:51:52.788607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:52.788570 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" event={"ID":"56500d5e-8f5f-4012-8077-e98730a4ab62","Type":"ContainerStarted","Data":"c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f"} Nov 02 21:51:52.902059 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:52.902003 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podStartSLOduration=2.901990107 podStartE2EDuration="2.901990107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:52.900247993 +0000 UTC m=+1051.861798312" watchObservedRunningTime="2026-11-02 21:51:52.901990107 +0000 UTC m=+1051.863540426" Nov 02 21:51:53.196841 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:53.196817 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:51:54.510928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:54.510890 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:56.024617 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:56.024586 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:59.213216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:51:59.213178 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:01.114641 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:01.114602 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:01.214223 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:01.214189 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:52:01.221380 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:01.221352 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:52:01.868032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:01.868000 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:52:03.931698 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:03.931666 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.173142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:08.173097 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.325076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:08.325049 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.475187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:08.475155 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:08.632495 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:08.632454 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:13.311150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:13.311102 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:15.994968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:15.994933 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:16.005521 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:52:16.005487 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:52:18.001586 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:18.001553 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:22.806994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:22.806954 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:27.495865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:27.495383 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:32.204155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:32.204105 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:32.561627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:32.561549 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:37.007036 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:37.007000 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:39.842869 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:39.842830 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:41.705880 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:41.705840 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:45.489681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:45.489645 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:45.902939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:45.902856 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:46.005059 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:52:46.005026 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:52:46.507594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:46.507556 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:51.259482 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:51.259442 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:55.907723 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:55.907686 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:58.432171 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:58.432138 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:58.582985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:52:58.582946 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:00.598535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.598505 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:00.618630 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.618599 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6"] Nov 02 21:53:00.645325 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.645285 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.707995 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.707227 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6"] Nov 02 21:53:00.713989 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.713948 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.714197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.714177 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.714284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.714215 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.714346 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.714312 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv62m\" (UniqueName: \"kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.714385 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.714369 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.714473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.714456 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.730237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.730214 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-10fdd5\"/\"distinct-2-w4-10fdd5-kserve-self-signed-certs\"" Nov 02 21:53:00.748099 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.748065 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:00.819190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.819152 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.820816 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.820786 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.821364 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.821345 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.822214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.820636 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.822214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.821536 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv62m\" (UniqueName: \"kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.822214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.821636 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.822214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.821733 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.822214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.822084 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.825152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.823244 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.826020 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.825992 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.831072 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.831038 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:00.878047 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:00.877991 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv62m\" (UniqueName: \"kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m\") pod \"distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:01.000411 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:01.000368 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:01.438951 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:01.438920 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6"] Nov 02 21:53:01.440678 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:53:01.440635 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a13596_0a3e_45e4_ab6e_25c06f8abb13.slice/crio-c3d3fe2f28ac20d9cbbeaddb38ca5217bf1e367967c0fc7a136e7bffa6b52aaf WatchSource:0}: Error finding container c3d3fe2f28ac20d9cbbeaddb38ca5217bf1e367967c0fc7a136e7bffa6b52aaf: Status 404 returned error can't find the container with id c3d3fe2f28ac20d9cbbeaddb38ca5217bf1e367967c0fc7a136e7bffa6b52aaf Nov 02 21:53:02.229515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:02.229475 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" event={"ID":"e6a13596-0a3e-45e4-ab6e-25c06f8abb13","Type":"ContainerStarted","Data":"335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb"} Nov 02 21:53:02.229515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:02.229519 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" event={"ID":"e6a13596-0a3e-45e4-ab6e-25c06f8abb13","Type":"ContainerStarted","Data":"c3d3fe2f28ac20d9cbbeaddb38ca5217bf1e367967c0fc7a136e7bffa6b52aaf"} Nov 02 21:53:02.344655 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:02.344603 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podStartSLOduration=2.344589102 podStartE2EDuration="2.344589102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:02.341177415 +0000 UTC m=+1121.302727732" watchObservedRunningTime="2026-11-02 21:53:02.344589102 +0000 UTC m=+1121.306139421" Nov 02 21:53:03.001471 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:03.001247 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:05.304709 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:05.304618 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:08.544826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:08.544785 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:10.007431 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:10.007396 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:11.000979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:11.000942 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:11.013715 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:11.013692 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:11.303493 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:11.303417 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:53:14.717157 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:14.717090 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:16.006174 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:53:16.006115 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:53:19.410408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:19.410365 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:24.095899 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:24.095865 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:26.895714 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:26.895680 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p"] Nov 02 21:53:26.899605 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:26.899587 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063418 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063495 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fsf\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063613 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063661 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063745 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063846 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063834 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063853 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.063947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.063869 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.064023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.064001 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.103070 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.103041 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p"] Nov 02 21:53:27.164749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.164667 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.164749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.164705 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.164749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.164729 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165014 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.164817 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165014 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.164857 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165014 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.164944 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fsf\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165082 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165104 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165185 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165201 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165403 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165247 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165413 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165515 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165478 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.165763 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.165739 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.167251 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.167229 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.167360 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.167322 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.221940 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.221910 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.222094 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.222033 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fsf\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf\") pod \"e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.512198 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.512164 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:27.951336 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:53:27.951292 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bb2b3d_8bbe_4709_8f4a_637b7bc2ca43.slice/crio-d41dfe5cf7e6bd6acea784f793f791daf1b4b6febb03d6cd2308e1e7fd6eafa3 WatchSource:0}: Error finding container d41dfe5cf7e6bd6acea784f793f791daf1b4b6febb03d6cd2308e1e7fd6eafa3: Status 404 returned error can't find the container with id d41dfe5cf7e6bd6acea784f793f791daf1b4b6febb03d6cd2308e1e7fd6eafa3 Nov 02 21:53:27.954161 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.954084 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:27.954276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.954251 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:27.954366 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.954350 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:27.955291 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:27.955088 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p"] Nov 02 21:53:28.438767 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:28.438664 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" event={"ID":"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43","Type":"ContainerStarted","Data":"056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3"} Nov 02 21:53:28.438767 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:28.438740 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" event={"ID":"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43","Type":"ContainerStarted","Data":"d41dfe5cf7e6bd6acea784f793f791daf1b4b6febb03d6cd2308e1e7fd6eafa3"} Nov 02 21:53:28.512846 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:28.512814 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:28.605703 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:28.605642 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podStartSLOduration=2.605629743 podStartE2EDuration="2.605629743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:28.602192118 +0000 UTC m=+1147.563742439" watchObservedRunningTime="2026-11-02 21:53:28.605629743 +0000 UTC m=+1147.567180062" Nov 02 21:53:28.808463 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:28.808426 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:29.532148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:29.531274 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:53:29.532148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:29.531353 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 21:53:30.517149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:30.515404 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:53:30.517149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:30.515461 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 21:53:31.529148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:31.528106 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:53:31.529148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:31.528180 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 21:53:32.520150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:32.516458 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:53:32.520150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:32.516530 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 21:53:33.538149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:33.537970 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:53:33.538149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:33.538026 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 21:53:33.620149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:33.619507 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:34.512936 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:34.512903 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:53:34.513158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:34.512962 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 21:53:35.518928 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:35.518889 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:35.520029 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:35.520002 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:35.521659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:35.521640 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 21:53:38.263923 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:38.263888 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:39.925531 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:39.925489 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2"] Nov 02 21:53:39.929758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:39.929734 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:39.987825 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:39.987788 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2"] Nov 02 21:53:40.032598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.032571 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"openshift-service-ca.crt\"" Nov 02 21:53:40.032598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.032586 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"kube-root-ca.crt\"" Nov 02 21:53:40.032598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.032575 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"default-dockercfg-xxmgz\"" Nov 02 21:53:40.032856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.032804 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-96c052\"/\"distinct-2-w5-96c052-kserve-self-signed-certs\"" Nov 02 21:53:40.040745 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.040722 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:40.101157 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.078924 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npll9\" (UniqueName: \"kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.101481 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.101452 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.127149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.105253 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.127574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.127535 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.145564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.145521 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.145744 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.145624 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247336 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247298 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npll9\" (UniqueName: \"kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247528 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247352 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247528 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247401 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247528 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247500 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247589 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247700 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247679 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247834 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247812 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247952 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247864 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.247997 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.247962 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.249735 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.249712 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.250000 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.249981 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.325598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.325371 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npll9\" (UniqueName: \"kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9\") pod \"distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.542349 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.542266 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:40.913437 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:40.913414 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2"] Nov 02 21:53:40.914704 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:53:40.914673 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde78196_f98e_444a_9419_25e38531db04.slice/crio-6f9a0d2e83eb056907b9ba322e574e3a3cf35d8b98a2f51d2c9ed4f8ec10692b WatchSource:0}: Error finding container 6f9a0d2e83eb056907b9ba322e574e3a3cf35d8b98a2f51d2c9ed4f8ec10692b: Status 404 returned error can't find the container with id 6f9a0d2e83eb056907b9ba322e574e3a3cf35d8b98a2f51d2c9ed4f8ec10692b Nov 02 21:53:41.533206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:41.533170 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" event={"ID":"fde78196-f98e-444a-9419-25e38531db04","Type":"ContainerStarted","Data":"354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf"} Nov 02 21:53:41.533602 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:41.533212 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" event={"ID":"fde78196-f98e-444a-9419-25e38531db04","Type":"ContainerStarted","Data":"6f9a0d2e83eb056907b9ba322e574e3a3cf35d8b98a2f51d2c9ed4f8ec10692b"} Nov 02 21:53:41.542491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:41.542469 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:41.647829 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:41.647771 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podStartSLOduration=2.6477563440000003 podStartE2EDuration="2.647756344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:41.645908468 +0000 UTC m=+1160.607458789" watchObservedRunningTime="2026-11-02 21:53:41.647756344 +0000 UTC m=+1160.609306714" Nov 02 21:53:42.330979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.330939 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7"] Nov 02 21:53:42.334952 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.334933 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375025 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375239 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375069 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375239 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375166 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375361 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375277 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375361 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375313 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp45x\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375472 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375394 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375472 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375442 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375489 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.375576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.375532 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477019 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.476986 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477218 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477046 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477218 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477110 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477218 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477193 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477225 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp45x\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477320 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477395 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477421 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477651 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477426 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477651 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477512 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477733 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477753 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.477983 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.477960 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.478321 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.478296 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.479963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.479941 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.480200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.480181 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.534440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.534407 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp45x\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.539104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.537445 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.539676 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.539650 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7"] Nov 02 21:53:42.653010 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.652940 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:42.999931 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:42.999897 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:43.088439 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.086404 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7"] Nov 02 21:53:43.095074 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:53:43.094864 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681cba6a_9e23_47a6_9aef_f2ffbd3fbd87.slice/crio-1e11d2fcc4b221f4c44d5422d969ec1a5db4806fa1f088c76020d79c93e17b08 WatchSource:0}: Error finding container 1e11d2fcc4b221f4c44d5422d969ec1a5db4806fa1f088c76020d79c93e17b08: Status 404 returned error can't find the container with id 1e11d2fcc4b221f4c44d5422d969ec1a5db4806fa1f088c76020d79c93e17b08 Nov 02 21:53:43.099706 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.099668 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:43.099854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.099751 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:43.099854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.099799 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:43.554134 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.554081 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" event={"ID":"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87","Type":"ContainerStarted","Data":"7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc"} Nov 02 21:53:43.554506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.554175 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" event={"ID":"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87","Type":"ContainerStarted","Data":"1e11d2fcc4b221f4c44d5422d969ec1a5db4806fa1f088c76020d79c93e17b08"} Nov 02 21:53:43.653189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.653155 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:43.781072 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:43.781021 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podStartSLOduration=1.7810060650000001 podStartE2EDuration="1.781006065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:43.728602199 +0000 UTC m=+1162.690152521" watchObservedRunningTime="2026-11-02 21:53:43.781006065 +0000 UTC m=+1162.742556457" Nov 02 21:53:44.655013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:44.654976 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:53:44.655427 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:44.655029 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 21:53:45.653218 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:45.653183 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:53:45.653397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:45.653261 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 21:53:46.005809 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:53:46.005775 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:53:46.653396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:46.653357 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:53:46.653577 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:46.653433 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 21:53:47.653478 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:47.653442 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:53:47.653853 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:47.653501 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 21:53:47.838890 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:47.838855 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:48.654044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:48.654005 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:53:48.654462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:48.654080 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 21:53:49.654104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:49.654067 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:53:49.654500 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:49.654159 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 21:53:50.542631 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:50.542590 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:50.559950 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:50.559923 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:50.607083 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:50.607055 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:53:50.657884 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:50.657858 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:50.658273 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:50.658197 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:50.658975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:50.658952 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 21:53:52.539922 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:52.539883 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:57.253322 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:53:57.253281 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:01.908629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:01.908594 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:06.608904 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:06.608871 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:08.876202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:08.876175 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:11.299403 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:11.299369 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:12.855948 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.855914 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf"] Nov 02 21:54:12.860193 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.860172 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899089 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899066 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4hl\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899120 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899227 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899254 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899362 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899381 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899476 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899544 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899530 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.899603 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.899591 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:12.968739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:12.968705 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf"] Nov 02 21:54:13.000288 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000260 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000295 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000425 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000468 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000578 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000734 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000713 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4hl\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000794 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000768 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000815 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000814 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.000964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000870 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.001025 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.000980 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.001146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.001094 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.001221 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.001160 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.001321 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.001294 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.002607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.002583 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.003012 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.002995 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.056517 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.056486 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.056659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.056641 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4hl\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.173904 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.173831 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h"] Nov 02 21:54:13.174297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.174266 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:13.178622 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.178586 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.231646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.231602 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-96c052-openshift-default-dockercfg-fh882\"" Nov 02 21:54:13.234054 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.234031 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h"] Nov 02 21:54:13.305258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305110 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305215 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305258 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305333 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfdg\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305354 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305396 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305410 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305493 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.305578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.305514 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407064 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407024 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407068 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407210 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407248 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407283 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407349 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfdg\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407458 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407635 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407584 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407644 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407673 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407755 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.407807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.407790 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.408044 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.408021 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.408099 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.408036 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.409632 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.409614 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.409946 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.409930 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.463412 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.463385 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.463553 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.463524 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfdg\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.505186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.505165 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:13.606534 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.604981 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf"] Nov 02 21:54:13.609344 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.609299 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:13.609458 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.609363 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:13.609458 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.609395 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:13.756085 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.755990 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" event={"ID":"20b9e799-217d-4d17-b996-a387a1e17101","Type":"ContainerStarted","Data":"b1c593dfb9f984335cb37303c886a788cd6b4a745ce926eef2e22a82be952265"} Nov 02 21:54:13.756085 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.756070 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" event={"ID":"20b9e799-217d-4d17-b996-a387a1e17101","Type":"ContainerStarted","Data":"d070f678e176675bf09b3b157cf198fc32481ec871c18e657c8665f518a336b2"} Nov 02 21:54:13.874240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.874158 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podStartSLOduration=1.874108229 podStartE2EDuration="1.874108229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:13.87209159 +0000 UTC m=+1192.833641902" watchObservedRunningTime="2026-11-02 21:54:13.874108229 +0000 UTC m=+1192.835658552" Nov 02 21:54:13.887901 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:54:13.887872 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6020c3f1_0598_405e_af0f_90bd7b629e1e.slice/crio-3eb47648f4e05628dc741128d3bd11a6ab98e49bca7d901b9f7ebe9282c5768d WatchSource:0}: Error finding container 3eb47648f4e05628dc741128d3bd11a6ab98e49bca7d901b9f7ebe9282c5768d: Status 404 returned error can't find the container with id 3eb47648f4e05628dc741128d3bd11a6ab98e49bca7d901b9f7ebe9282c5768d Nov 02 21:54:13.888977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.888944 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h"] Nov 02 21:54:13.890584 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.890550 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:13.890650 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.890633 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:13.890689 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:13.890677 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:14.174723 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:14.174642 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:14.176154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:14.176100 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:14.176282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:14.176158 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:14.765248 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:14.765207 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" event={"ID":"6020c3f1-0598-405e-af0f-90bd7b629e1e","Type":"ContainerStarted","Data":"492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736"} Nov 02 21:54:14.765479 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:14.765275 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" event={"ID":"6020c3f1-0598-405e-af0f-90bd7b629e1e","Type":"ContainerStarted","Data":"3eb47648f4e05628dc741128d3bd11a6ab98e49bca7d901b9f7ebe9282c5768d"} Nov 02 21:54:14.889144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:14.888639 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podStartSLOduration=1.888624306 podStartE2EDuration="1.888624306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:14.886730958 +0000 UTC m=+1193.848281277" watchObservedRunningTime="2026-11-02 21:54:14.888624306 +0000 UTC m=+1193.850174626" Nov 02 21:54:15.180863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:15.180759 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:15.180863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:15.180825 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:15.505968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:15.505932 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:15.507755 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:15.507729 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:54:15.507817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:15.507778 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 21:54:16.006951 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:54:16.006919 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:54:16.125159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:16.123602 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:16.184149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:16.183283 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:16.184149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:16.183347 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:16.400399 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:16.399562 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:16.583150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:16.567556 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:54:16.583150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:16.567619 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 21:54:17.175782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:17.175745 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:17.176300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:17.175807 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:17.531149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:17.529946 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:54:17.531149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:17.530015 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 21:54:18.177152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:18.176591 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:18.177152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:18.176643 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:18.505516 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:18.505474 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:54:18.505687 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:18.505545 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 21:54:19.176761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:19.176422 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:19.176761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:19.176488 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:19.506088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:19.506028 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:54:19.506629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:19.506097 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 21:54:20.174598 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:20.174555 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:54:20.174807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:20.174605 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 21:54:20.505580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:20.505535 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:54:20.505760 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:20.505601 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 02 21:54:20.907964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:20.907881 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:21.498439 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:21.498404 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:21.499334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:21.499311 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:21.500232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:21.500208 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 21:54:22.076148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:22.075275 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:22.076622 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:22.076510 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:22.078370 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:22.078310 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 21:54:22.399568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:22.399447 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:24.056786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:24.056747 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:25.615841 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:25.615812 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:30.284840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:30.284808 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:31.429388 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:31.419579 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:35.049301 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:35.049196 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:39.702250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:39.702205 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:40.856548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:40.856519 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:44.354472 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:44.354425 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:46.006392 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:54:46.006359 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:54:49.179512 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:49.179441 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:49.892319 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:49.890559 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:54:49.892701 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:49.892654 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.009198 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.008603 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.164205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.158227 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.291891 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.291849 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:54:50.305367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.305324 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.523661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.523602 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.601208 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.600210 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.679580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.679260 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:54:50.679580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.679322 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.896839 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.896698 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:50.898840 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:50.897371 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:54:51.025156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.021584 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:51.026151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.026106 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:54:51.207213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.160221 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:51.207213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.175696 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:54:51.425177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.425148 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:54:51.426901 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.426880 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:54:51.432332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:51.431415 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:54:54.118336 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:54.118295 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:58.843154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:54:58.830013 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:03.503470 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:03.503432 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:08.208401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:08.208365 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:09.220802 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:09.220756 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:09.399371 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:09.399341 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:11.792773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.792725 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx"] Nov 02 21:55:11.799036 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.799007 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.905757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.905718 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.905920 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.905786 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.905964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.905910 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.905964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.905944 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.906029 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.905976 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.906029 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.906015 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.906147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.906133 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.906189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.906155 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.906277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.906265 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q475r\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:11.919152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:11.919095 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx"] Nov 02 21:55:12.007149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007096 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007194 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007217 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007240 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007257 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007313 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007298 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007318 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007430 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q475r\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007488 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007721 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007663 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.007783 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.007731 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.008038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.008018 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.008307 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.008288 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.008351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.008310 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.009497 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.009477 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.009636 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.009621 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.064027 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.063944 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.064027 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.063982 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q475r\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.112899 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.112870 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:12.308645 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.308614 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w"] Nov 02 21:55:12.312926 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.312900 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.380186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.379698 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w"] Nov 02 21:55:12.413373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413338 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413536 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413386 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413536 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413417 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413611 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413530 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413611 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413582 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413673 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz6f\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413783 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413767 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.413952 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.413936 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.414063 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.414049 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515250 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515401 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515435 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515461 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515486 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515519 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515557 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515596 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz6f\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515664 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.515924 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.515854 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.516397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.516371 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.516709 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.516623 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.516820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.516746 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.517031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.517006 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.518344 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.518319 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.518881 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.518609 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.573342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.573268 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.573512 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.573495 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz6f\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.592719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.592694 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx"] Nov 02 21:55:12.593594 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:12.593570 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe5f8f9_9acd_4919_9b79_213f822f8dbf.slice/crio-8f832c8c756c346a2216e84ab3924ad2f24614a0b768186fcaa10d85c2a4c808 WatchSource:0}: Error finding container 8f832c8c756c346a2216e84ab3924ad2f24614a0b768186fcaa10d85c2a4c808: Status 404 returned error can't find the container with id 8f832c8c756c346a2216e84ab3924ad2f24614a0b768186fcaa10d85c2a4c808 Nov 02 21:55:12.595845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.595812 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.595919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.595876 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.595919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.595910 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:12.630015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.629988 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:12.903532 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:12.903466 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:13.051428 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.051396 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh"] Nov 02 21:55:13.055419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.055391 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.068733 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.068686 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w"] Nov 02 21:55:13.126863 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.126821 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.127108 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.127087 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.127316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.127288 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.129297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.128948 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh"] Nov 02 21:55:13.139453 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139429 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139494 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139526 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139555 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139574 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltq24\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139794 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139645 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139794 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139669 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139794 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139703 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.139794 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.139756 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241092 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241169 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241215 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241272 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241335 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241434 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241480 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.241545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241534 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.242071 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.241563 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltq24\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.242234 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.242210 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.242302 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.242219 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.242592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.242565 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.242735 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.242718 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.243020 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.243000 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.245901 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.245876 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.254616 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.254589 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.298601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.298566 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.298845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.298822 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltq24\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24\") pod \"e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.372949 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.372914 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:13.563189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.562982 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" event={"ID":"cbe5f8f9-9acd-4919-9b79-213f822f8dbf","Type":"ContainerStarted","Data":"f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875"} Nov 02 21:55:13.563189 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.563055 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" event={"ID":"cbe5f8f9-9acd-4919-9b79-213f822f8dbf","Type":"ContainerStarted","Data":"8f832c8c756c346a2216e84ab3924ad2f24614a0b768186fcaa10d85c2a4c808"} Nov 02 21:55:13.570365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.569860 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" event={"ID":"ed2f2a53-073d-4a89-98cc-9ccd2618b734","Type":"ContainerStarted","Data":"89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0"} Nov 02 21:55:13.570365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.569921 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" event={"ID":"ed2f2a53-073d-4a89-98cc-9ccd2618b734","Type":"ContainerStarted","Data":"8360c4ad3a87ff0347a8f5c3e94064d7a0e775a38c4770c4fb147d117d4b7ddf"} Nov 02 21:55:13.644149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.641325 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:13.705583 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.705532 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podStartSLOduration=2.7055166269999997 podStartE2EDuration="2.705516627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:13.702831617 +0000 UTC m=+1252.664381943" watchObservedRunningTime="2026-11-02 21:55:13.705516627 +0000 UTC m=+1252.667066951" Nov 02 21:55:13.836150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.830440 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podStartSLOduration=1.8304196 podStartE2EDuration="1.8304196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:13.827685854 +0000 UTC m=+1252.789236175" watchObservedRunningTime="2026-11-02 21:55:13.8304196 +0000 UTC m=+1252.791969921" Nov 02 21:55:13.860427 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.860376 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh"] Nov 02 21:55:13.861695 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:13.861654 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e9a3e8_77ae_41ab_b9b2_44acdd0697bd.slice/crio-63a77c329a6dad5a91f70eab9b2fde9845152018034a8b5c88e43320c31a0d81 WatchSource:0}: Error finding container 63a77c329a6dad5a91f70eab9b2fde9845152018034a8b5c88e43320c31a0d81: Status 404 returned error can't find the container with id 63a77c329a6dad5a91f70eab9b2fde9845152018034a8b5c88e43320c31a0d81 Nov 02 21:55:13.864716 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.864686 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.864819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.864754 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:13.864819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:13.864784 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:14.120175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.117966 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:14.120175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.118030 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:14.120175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.118324 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:14.602560 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.602515 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" event={"ID":"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd","Type":"ContainerStarted","Data":"ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b"} Nov 02 21:55:14.602762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.602615 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" event={"ID":"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd","Type":"ContainerStarted","Data":"63a77c329a6dad5a91f70eab9b2fde9845152018034a8b5c88e43320c31a0d81"} Nov 02 21:55:14.637022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.636982 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:14.637455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:14.637419 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:15.116616 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.116575 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:15.116909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.116876 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:15.390969 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.389876 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:15.404149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.393315 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:15.404149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.393384 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:15.651210 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.650395 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:15.651210 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:15.650470 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:16.103364 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:55:16.103322 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:55:16.114555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:16.114507 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:16.114817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:16.114790 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:16.373536 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:16.373446 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:16.373818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:16.373790 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:16.643206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:16.642579 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:16.643206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:16.642643 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:17.120150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:17.117725 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:17.120150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:17.117804 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:17.375209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:17.374581 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:17.375209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:17.374647 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:17.632209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:17.631568 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:17.632209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:17.631635 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:18.120159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.116659 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:18.120159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.116742 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:18.175276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.162394 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:18.406668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.406567 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:18.406931 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.406896 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:18.632162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.631571 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:18.632162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:18.631635 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:19.113537 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.113497 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:19.113820 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.113789 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:19.378074 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.377960 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:19.381826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.381709 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:19.381826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.381761 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:19.630790 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.630688 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:19.631069 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:19.631023 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:20.115182 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.115025 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:20.115182 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.115112 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:20.305351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.305265 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:20.378658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.378560 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:20.379282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.379232 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:20.389267 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.389224 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:20.553151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.541210 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:55:20.631614 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.631516 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:20.631917 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.631885 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:20.870885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:20.870832 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:55:21.259809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.259766 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:21.284547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.284507 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:21.335194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.299239 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.335194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.316035 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.335194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.316489 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:55:21.391150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.387701 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:21.592530 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.592422 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:21.597273 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.597233 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.631977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.438928 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.632319 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.464358 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.632502 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.561429 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.632643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.569530 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.632821 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.420352 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:21.633076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.633047 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:55:21.646503 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.630840 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:21.646673 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.438530 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.646766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.646559 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:21.646930 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.646896 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:55:21.726282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.726236 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:21.726589 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.726553 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.896254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.893725 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:21.896254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:21.895081 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:55:22.059533 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.059463 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podStartSLOduration=9.059439733 podStartE2EDuration="9.059439733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:14.747358184 +0000 UTC m=+1253.708908505" watchObservedRunningTime="2026-11-02 21:55:22.059439733 +0000 UTC m=+1261.020990056" Nov 02 21:55:22.129868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.129791 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 02 21:55:22.131146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.131087 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:55:22.131401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.131342 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 02 21:55:22.340312 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.340245 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.469054 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.340577 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:55:22.497165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.470197 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:55:22.497165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.370362 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 02 21:55:22.541868 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.541833 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:22.542140 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.542099 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:22.554273 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.554240 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:22.554416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.554302 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.658526 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.658408 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.669507 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.669410 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:55:22.702270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.702233 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:22.706150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.702713 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:22.706469 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.706439 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.727200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.727152 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:22.728905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.728875 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:55:22.764924 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.764896 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:55:22.772964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.772919 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:55:22.792003 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.791972 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:55:22.920285 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:22.919840 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.228207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.181350 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:23.231338 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.231285 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.577418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.577311 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.593148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.584268 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:55:23.717207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.673092 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:23.717207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.673178 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:23.717207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.692889 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:23.717207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.692955 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:23.718226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.718174 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.726592 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.726460 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:23.735173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.729098 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.735173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.729489 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.796444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.796401 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:23.965205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:23.953316 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:55:24.109111 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.109072 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:55:24.181720 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.181692 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:55:24.181979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.181966 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:55:24.212372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.212341 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:55:24.286017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.282302 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:24.286017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.282369 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:24.286017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.284626 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:55:24.314469 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.314440 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:55:24.506693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.434636 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:24.506693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.434692 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:24.598907 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.598797 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:24.669370 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.669317 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:24.684591 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.684557 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:55:24.684898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.684831 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 21:55:24.738312 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.722941 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:24.920203 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:24.902418 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:25.109353 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.109305 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:25.189648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.187528 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:25.189648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.188795 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:25.372241 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.337195 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 21:55:25.377145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.375484 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:55:25.377145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.375538 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 21:55:25.478147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.477238 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:55:25.686263 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.686216 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:25.702155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.686782 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:25.790235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:25.773482 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:55:26.191519 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.191422 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:26.192907 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.192884 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:26.193296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.193276 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 21:55:26.851905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.851870 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc"] Nov 02 21:55:26.856965 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.856949 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.918298 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.918260 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc"] Nov 02 21:55:26.932823 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.932798 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.932980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.932834 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.932980 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.932853 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.933095 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.933063 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.933237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.933220 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.933300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.933251 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.933300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.933275 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsvj\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.933430 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.933404 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:26.933504 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:26.933488 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.034835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.034790 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.034895 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035105 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035045 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035105 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035081 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsvj\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035326 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035500 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035407 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035572 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035537 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035636 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035579 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035630 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.035807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.035784 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.036177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.036149 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.036177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.036166 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.036414 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.036394 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.037475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.037450 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.037877 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.037851 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.039308 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.039285 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.090909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.090882 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsvj\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.090909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.090898 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token\") pod \"e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.172377 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.172279 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:27.415321 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.415281 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:27.415491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.415355 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:27.610438 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.610411 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc"] Nov 02 21:55:27.611921 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:27.611895 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b382c8_0005_4373_8a93_cd6d2ed4212a.slice/crio-4e5ee9b4c51eb6cd05db7eb252189077907fb7c9d240c024eef4393c930aed54 WatchSource:0}: Error finding container 4e5ee9b4c51eb6cd05db7eb252189077907fb7c9d240c024eef4393c930aed54: Status 404 returned error can't find the container with id 4e5ee9b4c51eb6cd05db7eb252189077907fb7c9d240c024eef4393c930aed54 Nov 02 21:55:27.614939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.614902 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:27.615023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.614992 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:27.615061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.615038 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:27.764172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.764137 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:27.764942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.764919 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:27.766593 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:27.766571 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 21:55:28.091563 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.091520 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" event={"ID":"d7b382c8-0005-4373-8a93-cd6d2ed4212a","Type":"ContainerStarted","Data":"c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af"} Nov 02 21:55:28.092029 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.091601 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" event={"ID":"d7b382c8-0005-4373-8a93-cd6d2ed4212a","Type":"ContainerStarted","Data":"4e5ee9b4c51eb6cd05db7eb252189077907fb7c9d240c024eef4393c930aed54"} Nov 02 21:55:28.173026 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.172992 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:28.174478 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.174452 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:28.174635 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.174498 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:28.211043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.210979 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podStartSLOduration=2.210960021 podStartE2EDuration="2.210960021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:28.206358867 +0000 UTC m=+1267.167909183" watchObservedRunningTime="2026-11-02 21:55:28.210960021 +0000 UTC m=+1267.172510346" Nov 02 21:55:28.443261 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.443225 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2"] Nov 02 21:55:28.448340 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.448317 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.555187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.555151 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2"] Nov 02 21:55:28.559998 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.559966 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560133 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560047 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560188 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560499 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560447 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560612 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560500 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cd5k\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560612 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560575 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560705 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560648 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560705 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560696 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.560943 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.560914 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.662927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.662887 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.662948 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663013 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663048 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cd5k\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663142 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663077 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663169 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663226 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663295 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.663421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.663387 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.664306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.664278 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.664569 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.664551 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.664784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.664764 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.665564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.665537 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.666852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.666792 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.668979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.668949 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.669484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.669460 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.748433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.747296 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.748433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.747705 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cd5k\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:28.764242 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:28.764021 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:29.172968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.172720 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:29.172968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.172834 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:29.244535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.244498 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2"] Nov 02 21:55:29.257535 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:29.257496 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6c42fc_46b8_4c30_9129_691f98079984.slice/crio-b264f6f7728d50f296907906718901cd23d4303e205186f5d111e55b2d7780e2 WatchSource:0}: Error finding container b264f6f7728d50f296907906718901cd23d4303e205186f5d111e55b2d7780e2: Status 404 returned error can't find the container with id b264f6f7728d50f296907906718901cd23d4303e205186f5d111e55b2d7780e2 Nov 02 21:55:29.260519 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.260479 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:29.260620 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.260566 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:29.260620 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.260608 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:29.447055 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:29.446709 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:30.106119 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.106079 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" event={"ID":"0e6c42fc-46b8-4c30-9129-691f98079984","Type":"ContainerStarted","Data":"741e4bf961bd10f032e0c9e377d835204d91f7e5395dc092c5f5d5253ce927df"} Nov 02 21:55:30.106330 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.106159 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" event={"ID":"0e6c42fc-46b8-4c30-9129-691f98079984","Type":"ContainerStarted","Data":"b264f6f7728d50f296907906718901cd23d4303e205186f5d111e55b2d7780e2"} Nov 02 21:55:30.177147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.175519 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:30.177147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.175579 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:30.292828 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.292776 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podStartSLOduration=2.292762153 podStartE2EDuration="2.292762153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:30.272602212 +0000 UTC m=+1269.234152533" watchObservedRunningTime="2026-11-02 21:55:30.292762153 +0000 UTC m=+1269.254312530" Nov 02 21:55:30.764784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.764731 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:30.767692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.767655 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:55:30.767839 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:30.767728 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 21:55:31.165565 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:31.165473 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:31.173334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:31.173298 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:31.173458 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:31.173356 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:31.369775 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:31.369729 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:31.769148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:31.768505 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:55:31.769148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:31.768578 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 21:55:32.172903 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:32.172801 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:32.172903 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:32.172864 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:32.767523 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:32.767486 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:55:32.768075 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:32.768048 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 21:55:33.184204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:33.175474 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:33.184204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:33.175534 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:33.764587 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:33.764555 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:55:33.764769 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:33.764630 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 21:55:34.172977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:34.172891 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:55:34.172977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:34.172946 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 21:55:34.440773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:34.440676 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:34.764944 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:34.764907 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:55:34.765154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:34.764957 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 21:55:35.364661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:35.363567 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:35.365195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:35.364939 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:35.368342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:35.368299 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 21:55:35.771438 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:35.771401 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:55:35.772256 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:35.772073 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 02 21:55:36.791670 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:36.791637 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:36.793055 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:36.793034 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:36.793844 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:36.793824 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 21:55:37.061201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:37.060378 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd"] Nov 02 21:55:37.065050 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:37.065018 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-546d97f9-g6qfd"] Nov 02 21:55:37.174766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:37.174731 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5642d0-5774-4422-825b-a7395048ab67" path="/var/lib/kubelet/pods/3b5642d0-5774-4422-825b-a7395048ab67/volumes" Nov 02 21:55:39.090979 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:39.090946 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:43.561725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.561693 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2"] Nov 02 21:55:43.654637 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.654602 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.709543 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.709502 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2"] Nov 02 21:55:43.728391 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.728353 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.729006 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.728984 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.752439 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.752408 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.752781 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.752766 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.759341 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.759318 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.759804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.759784 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.759966 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.759952 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgpb\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.760436 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.760420 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.760852 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.760818 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.934640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.934515 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.988763 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.988851 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.988880 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.988915 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.988979 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.989029 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.989067 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgpb\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.989157 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.989268 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:43.996147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:43.990044 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.032154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.028678 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.032154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.029440 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.032154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.031337 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.039144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.035275 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.085511 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.085477 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.096897 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.095480 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgpb\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.097284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.097264 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.218798 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.218768 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo\") pod \"e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:44.379913 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:44.379883 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:45.820174 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:45.772459 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:45.823096 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:45.822021 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:55:46.202755 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.202669 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.212679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.209117 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.212679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.210192 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:55:46.265426 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.265360 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.273887 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.273845 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2"] Nov 02 21:55:46.348257 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:46.348090 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f0ca1c_72dc_4d22_96e2_547800ce142f.slice/crio-8037e2053409c498fda8c4ef11899dc7561b50b6f391690799a95179bb8fd0de WatchSource:0}: Error finding container 8037e2053409c498fda8c4ef11899dc7561b50b6f391690799a95179bb8fd0de: Status 404 returned error can't find the container with id 8037e2053409c498fda8c4ef11899dc7561b50b6f391690799a95179bb8fd0de Nov 02 21:55:46.372369 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.371781 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:46.372369 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.371875 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:46.372369 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.371924 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:46.574261 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:55:46.569335 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:55:46.574261 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.572607 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" event={"ID":"e3f0ca1c-72dc-4d22-96e2-547800ce142f","Type":"ContainerStarted","Data":"8037e2053409c498fda8c4ef11899dc7561b50b6f391690799a95179bb8fd0de"} Nov 02 21:55:46.615569 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.614212 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.672150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.671656 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.699520 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.698880 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.778158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.775207 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:55:46.778158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.775551 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:55:46.824156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.822223 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:46.824156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:46.823694 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:55:47.233165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.211349 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.233165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.218831 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.233165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.219768 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.238474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.238445 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:55:47.238935 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.238915 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:55:47.239089 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.239079 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:55:47.277149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.274549 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.277149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.275742 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:55:47.709062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.681607 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:47.709062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.682866 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:55:47.713147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.710391 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:47.717480 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:47.717271 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:55:48.118206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.067458 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:55:48.183412 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.183366 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:55:48.225323 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.225279 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:55:48.441214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.418749 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:55:48.523506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.523457 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" podUID="615e6adb-9482-405a-85f3-98d3af5df4cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:48.523996 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.523952 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:48.524875 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.524848 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:55:48.834151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.808593 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:48.878404 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:48.868235 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:55:49.001302 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.001255 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.079150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.071780 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.233211 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.228194 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.236889 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.236848 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.427430 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.427384 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.525401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.525269 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.555424 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.543189 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:49.605327 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.605288 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" event={"ID":"e3f0ca1c-72dc-4d22-96e2-547800ce142f","Type":"ContainerStarted","Data":"7f4b6a58218d0512f5af4a0e0bc84d6136dd7e0ff4d18ebfb0b15ca3a3106502"} Nov 02 21:55:49.605618 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.547639 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:49.605829 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.605786 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:49.606049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.606026 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.863209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.859343 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:49.918154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:49.897270 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.013149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.005833 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podStartSLOduration=7.005811209 podStartE2EDuration="7.005811209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:49.635216801 +0000 UTC m=+1288.596767124" watchObservedRunningTime="2026-11-02 21:55:50.005811209 +0000 UTC m=+1288.967361530" Nov 02 21:55:50.122860 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.121233 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:55:50.132288 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.132251 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:50.144160 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.132512 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.283300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.283250 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.297222 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.284194 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.437242 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.432218 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.580259 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.580210 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.609158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.606333 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.707228 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.683592 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:50.707228 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.683656 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:50.708914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.708032 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:50.708914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.708091 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:50.901418 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.864203 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:50.962329 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:50.962213 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.140322 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.138265 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:55:51.303211 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.294202 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.503934 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.503886 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.588230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.564907 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:51.655845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.655802 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:51.701964 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.598973 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.702232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.601219 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:55:51.702619 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.619596 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:55:51.704220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.620517 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:55:51.705644 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.574423 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.705804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.615938 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:51.705956 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.705930 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.706095 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.688101 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:51.706372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.650429 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:51.706372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.706318 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.904204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.895231 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:55:51.904204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.895297 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:51.914643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.914593 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:55:51.967169 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:51.966210 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:52.101615 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.101572 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:52.155754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.155662 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:52.269421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.269391 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:55:52.342347 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.342282 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:52.407623 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.371307 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:55:52.700510 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.692230 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:52.700510 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.692300 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:52.761377 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.761073 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:52.762640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.762473 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:55:52.796860 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.796835 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:55:52.804119 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.804099 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:55:52.809448 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.809424 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:55:52.846106 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.840590 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:55:52.899909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.899886 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:55:52.969646 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:52.969573 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:55:53.263397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.263319 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct"] Nov 02 21:55:53.268523 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.268504 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.293628 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.293603 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.293761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.293743 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.293886 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.293871 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj59h\" (UniqueName: \"kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.359066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.359021 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct"] Nov 02 21:55:53.360060 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.360030 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:55:53.361642 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.361625 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 21:55:53.381808 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.381778 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:53.381940 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.381836 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:53.395065 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.395042 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj59h\" (UniqueName: \"kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.395208 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.395135 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.395285 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.395269 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.395931 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.395910 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.397652 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.397625 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.467801 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.466194 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj59h\" (UniqueName: \"kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h\") pod \"maas-api-revoke-sub-4580c2d6-4bsct\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:53.594019 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:53.593937 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:54.007334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:54.007303 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct"] Nov 02 21:55:54.008412 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:54.008385 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388fc395_c211_4f65_a6ac_9c23be0c6b91.slice/crio-b1df1d598e680791f67385c476915c96f2101b13dc77ee6570d65b3413d707ce WatchSource:0}: Error finding container b1df1d598e680791f67385c476915c96f2101b13dc77ee6570d65b3413d707ce: Status 404 returned error can't find the container with id b1df1d598e680791f67385c476915c96f2101b13dc77ee6570d65b3413d707ce Nov 02 21:55:54.111090 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:54.111060 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" event={"ID":"388fc395-c211-4f65-a6ac-9c23be0c6b91","Type":"ContainerStarted","Data":"b1df1d598e680791f67385c476915c96f2101b13dc77ee6570d65b3413d707ce"} Nov 02 21:55:54.382164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:54.382070 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:54.382164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:54.382120 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:55.175108 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:55.175063 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" event={"ID":"388fc395-c211-4f65-a6ac-9c23be0c6b91","Type":"ContainerStarted","Data":"5f217076373359ed3ac6a38f5d389cb1bc1e11e13168ae861e0a3fea136d2fd1"} Nov 02 21:55:55.291015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:55.290966 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" podStartSLOduration=2.290951539 podStartE2EDuration="2.290951539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:55.290327384 +0000 UTC m=+1294.251877704" watchObservedRunningTime="2026-11-02 21:55:55.290951539 +0000 UTC m=+1294.252501858" Nov 02 21:55:55.381694 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:55.381665 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:55.381835 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:55.381725 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:56.175938 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:56.175909 4285 generic.go:356] "Generic (PLEG): container finished" podID="388fc395-c211-4f65-a6ac-9c23be0c6b91" containerID="5f217076373359ed3ac6a38f5d389cb1bc1e11e13168ae861e0a3fea136d2fd1" exitCode=0 Nov 02 21:55:56.176429 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:56.176030 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" event={"ID":"388fc395-c211-4f65-a6ac-9c23be0c6b91","Type":"ContainerDied","Data":"5f217076373359ed3ac6a38f5d389cb1bc1e11e13168ae861e0a3fea136d2fd1"} Nov 02 21:55:56.235261 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:56.235224 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:56.388059 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:56.388022 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:56.388339 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:56.388309 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:57.356192 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.356163 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:57.381383 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.381355 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:55:57.381509 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.381415 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 21:55:57.385848 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.385824 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca\") pod \"388fc395-c211-4f65-a6ac-9c23be0c6b91\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " Nov 02 21:55:57.385962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.385949 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca\") pod \"388fc395-c211-4f65-a6ac-9c23be0c6b91\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " Nov 02 21:55:57.386087 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.386071 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h\" (UniqueName: \"kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h\") pod \"388fc395-c211-4f65-a6ac-9c23be0c6b91\" (UID: \"388fc395-c211-4f65-a6ac-9c23be0c6b91\") " Nov 02 21:55:57.386309 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.386283 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca" pod "388fc395-c211-4f65-a6ac-9c23be0c6b91" (UID: "388fc395-c211-4f65-a6ac-9c23be0c6b91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:57.386668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.386645 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:57.388006 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.387980 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca" pod "388fc395-c211-4f65-a6ac-9c23be0c6b91" (UID: "388fc395-c211-4f65-a6ac-9c23be0c6b91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:55:57.388149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.388107 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h" pod "388fc395-c211-4f65-a6ac-9c23be0c6b91" (UID: "388fc395-c211-4f65-a6ac-9c23be0c6b91"). InnerVolumeSpecName "kube-api-access-hj59h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:57.489084 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.489049 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388fc395-c211-4f65-a6ac-9c23be0c6b91-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:57.489084 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:57.489090 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj59h\" (UniqueName: \"kubernetes.io/projected/388fc395-c211-4f65-a6ac-9c23be0c6b91-kube-api-access-hj59h\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:55:58.193195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:58.193160 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" Nov 02 21:55:58.193390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:58.193160 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct" event={"ID":"388fc395-c211-4f65-a6ac-9c23be0c6b91","Type":"ContainerDied","Data":"b1df1d598e680791f67385c476915c96f2101b13dc77ee6570d65b3413d707ce"} Nov 02 21:55:58.193390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:58.193285 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1df1d598e680791f67385c476915c96f2101b13dc77ee6570d65b3413d707ce" Nov 02 21:55:58.480634 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:58.480557 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:58.482053 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:58.482032 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:58.484744 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:58.484725 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 21:55:59.028514 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.028483 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8"] Nov 02 21:55:59.029167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.029149 4285 cpu_manager.go:414] "containerMap: removing container" podUID="388fc395-c211-4f65-a6ac-9c23be0c6b91" containerName="revoke-keys" Nov 02 21:55:59.029220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.029168 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388fc395-c211-4f65-a6ac-9c23be0c6b91" containerName="revoke-keys" Nov 02 21:55:59.029279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.029270 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="388fc395-c211-4f65-a6ac-9c23be0c6b91" containerName="revoke-keys" Nov 02 21:55:59.032985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.032963 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.091164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.091136 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8"] Nov 02 21:55:59.111991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.111959 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112009 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112100 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112172 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112215 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112558 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112248 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112558 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112372 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112558 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112423 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg48j\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.112558 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.112528 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214407 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214376 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214419 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214446 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214470 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214485 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214552 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214597 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg48j\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214653 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.214906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.214853 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.215918 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.215385 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.215918 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.215433 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.215918 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.215867 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.216275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.216250 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.216409 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.216344 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.217276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.217256 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.217358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.217315 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.270238 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.270207 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.270516 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.270496 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg48j\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.347762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.347692 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:55:59.776996 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.776967 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8"] Nov 02 21:55:59.777563 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:55:59.777538 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288cffb2_d9ae_4278_ac1a_7a704291bd0a.slice/crio-70b0cdce588443598ced2868a5b5620ce3978ca44976a67ece63332af6f7e064 WatchSource:0}: Error finding container 70b0cdce588443598ced2868a5b5620ce3978ca44976a67ece63332af6f7e064: Status 404 returned error can't find the container with id 70b0cdce588443598ced2868a5b5620ce3978ca44976a67ece63332af6f7e064 Nov 02 21:55:59.780149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.780097 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:59.780220 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.780191 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:59.780256 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:55:59.780235 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:00.209376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:00.209336 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" event={"ID":"288cffb2-d9ae-4278-ac1a-7a704291bd0a","Type":"ContainerStarted","Data":"2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2"} Nov 02 21:56:00.209572 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:00.209399 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" event={"ID":"288cffb2-d9ae-4278-ac1a-7a704291bd0a","Type":"ContainerStarted","Data":"70b0cdce588443598ced2868a5b5620ce3978ca44976a67ece63332af6f7e064"} Nov 02 21:56:00.348421 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:00.348392 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:56:00.349563 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:00.349539 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:00.349681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:00.349581 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:00.377079 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:00.376996 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podStartSLOduration=2.376976518 podStartE2EDuration="2.376976518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:00.37274979 +0000 UTC m=+1299.334300109" watchObservedRunningTime="2026-11-02 21:56:00.376976518 +0000 UTC m=+1299.338526839" Nov 02 21:56:01.034174 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:01.033136 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:01.348601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:01.348520 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:01.348601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:01.348572 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:02.348919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:02.348876 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:02.349425 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:02.348940 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:03.349141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:03.348499 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:03.349141 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:03.348552 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:04.348426 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:04.348382 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:04.348643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:04.348433 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:05.209871 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:05.209833 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:05.348463 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:05.348423 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:05.348660 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:05.348478 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:05.385092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:05.385055 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:05.828396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:05.828350 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:05.987024 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:05.986983 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:06.348079 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:06.348042 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:56:06.348567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:06.348101 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 21:56:07.353968 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:07.353936 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:56:07.354481 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:07.354467 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:56:07.355179 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:07.355157 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 21:56:09.548462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:09.548423 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:10.514892 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:10.514855 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:11.481276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:11.481081 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:11.638983 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:11.638944 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:11.798795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:11.798708 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:11.969847 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:11.969810 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:12.293849 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.293814 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:12.553198 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.547362 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:12.639929 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.635657 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz"] Nov 02 21:56:12.646463 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.646431 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz"] Nov 02 21:56:12.646750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.646728 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.786163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.781507 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw"] Nov 02 21:56:12.787066 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.787041 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.848789 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.848715 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849143 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849351 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849549 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849412 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849549 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849526 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849660 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p6p\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849739 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849708 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849746 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.849791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.849774 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.962151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.954060 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.970181 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.966728 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.986273 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.986234 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.986552 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.986531 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.986679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.986662 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.986902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.986885 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.987008 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.986994 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.987159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.987142 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.995160 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.987320 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.995394 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.995369 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:12.995536 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.995518 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbh7p\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:12.995702 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:12.995685 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.014450 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.014409 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.014773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.014749 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.014987 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.014911 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p6p\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.015158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.015138 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.015301 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.015287 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.015427 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.015412 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.024149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.004993 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.024408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.005687 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.024535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.022647 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.024659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.024641 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5"] Nov 02 21:56:13.084154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.065181 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.084154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.065465 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.084154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.065936 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:13.084154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.082324 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.087316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.087292 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.099665 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.098890 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p6p\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135470 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135522 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135632 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135696 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135746 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135785 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135818 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbh7p\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135842 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135894 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135942 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135978 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdss8\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.135992 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136441 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136503 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136558 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136584 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136701 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136746 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.137351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.136773 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.143762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.143175 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.143762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.143431 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.144939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.144071 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.144939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.144139 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw"] Nov 02 21:56:13.144939 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.144503 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.148757 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.148670 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5"] Nov 02 21:56:13.158861 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.158801 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.159548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.159502 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.162231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.162184 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.163958 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.163931 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.215782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.215659 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbh7p\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257240 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257301 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257400 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257706 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257766 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257804 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257838 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.257941 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdss8\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.258056 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.258983 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.260151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.259316 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.270207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.261542 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.275448 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.272026 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.275448 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.272850 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.275448 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.273499 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.282155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.281937 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.297183 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.296260 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.317162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.317097 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.328955 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.328926 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdss8\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8\") pod \"e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.366021 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.365957 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:13.407108 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.407080 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:13.597881 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.597849 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:13.834456 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.827386 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz"] Nov 02 21:56:13.894742 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.874387 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw"] Nov 02 21:56:13.900359 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.900260 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.900689 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.900583 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.900956 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.900667 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.901212 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:56:13.901082 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3662e24_62ab_48c7_86ae_63a96df54380.slice/crio-cca1f9535f98898e98921c9f02a18e0fd9af4bb39bb351fd59e209211f44300d WatchSource:0}: Error finding container cca1f9535f98898e98921c9f02a18e0fd9af4bb39bb351fd59e209211f44300d: Status 404 returned error can't find the container with id cca1f9535f98898e98921c9f02a18e0fd9af4bb39bb351fd59e209211f44300d Nov 02 21:56:13.904994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.904763 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.904994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.904838 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.904994 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.904886 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:13.989275 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.989244 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj"] Nov 02 21:56:13.994200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:13.993592 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.017505 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.017478 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5"] Nov 02 21:56:14.033342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.029441 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:14.033342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.029540 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:14.033342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.029608 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.094034 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a16a02cb-gw-openshift-default-dockercfg-rwd2v\"" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.100587 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.100875 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.100997 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.101184 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.101245 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.101372 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.101611 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.101761 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.101907 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd89f\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.116180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.114161 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:14.127188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.123834 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj"] Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207218 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207282 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd89f\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207351 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207485 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207542 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207591 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207626 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207656 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.211693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.207742 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.276616 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.218465 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.324010 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.277792 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.344740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.279350 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.344740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.281419 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.344740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.281894 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.344740 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.340681 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.394146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.372748 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:14.394146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.390239 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.394146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.393525 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd89f\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.394146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.393531 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy\") pod \"e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.624177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.617438 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:14.661985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.661946 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" event={"ID":"298aee9e-ab9c-42af-a62c-ef8d48719bff","Type":"ContainerStarted","Data":"547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66"} Nov 02 21:56:14.662243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.662222 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" event={"ID":"298aee9e-ab9c-42af-a62c-ef8d48719bff","Type":"ContainerStarted","Data":"774d01f5d709ebf8cc5fd7600001804b031e4491e6012c00f1528ad7acf6a1b8"} Nov 02 21:56:14.664346 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.664318 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" event={"ID":"c3662e24-62ab-48c7-86ae-63a96df54380","Type":"ContainerStarted","Data":"cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215"} Nov 02 21:56:14.666886 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.666862 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" event={"ID":"c3662e24-62ab-48c7-86ae-63a96df54380","Type":"ContainerStarted","Data":"cca1f9535f98898e98921c9f02a18e0fd9af4bb39bb351fd59e209211f44300d"} Nov 02 21:56:14.668300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.668273 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" event={"ID":"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f","Type":"ContainerStarted","Data":"bacf9fd4d78c97717bd7691cf55086cea24b8d9f72ad13d12c4eaf5dfd9184f1"} Nov 02 21:56:14.668402 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.668322 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" event={"ID":"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f","Type":"ContainerStarted","Data":"6dfd3ac8314342de37daaa82fd4d94886972174ca7a0d88e194a0c9fc2fa2190"} Nov 02 21:56:14.787250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.783165 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podStartSLOduration=2.783108254 podStartE2EDuration="2.783108254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:14.781918707 +0000 UTC m=+1313.743469030" watchObservedRunningTime="2026-11-02 21:56:14.783108254 +0000 UTC m=+1313.744658575" Nov 02 21:56:14.901854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:14.901802 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podStartSLOduration=2.901786888 podStartE2EDuration="2.901786888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:14.898433887 +0000 UTC m=+1313.859984229" watchObservedRunningTime="2026-11-02 21:56:14.901786888 +0000 UTC m=+1313.863337208" Nov 02 21:56:15.048206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.047930 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podStartSLOduration=3.04791027 podStartE2EDuration="3.04791027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:15.04788286 +0000 UTC m=+1314.009433194" watchObservedRunningTime="2026-11-02 21:56:15.04791027 +0000 UTC m=+1314.009460593" Nov 02 21:56:15.085681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.085584 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj"] Nov 02 21:56:15.087895 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.087853 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:15.087981 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.087947 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:15.088016 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.087999 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:15.371207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.370501 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:15.375146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.372697 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:15.375146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.372756 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:15.407805 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.407772 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:15.410081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.410051 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:15.410224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.410142 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:15.613157 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.607887 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:15.613586 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.613558 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:15.613793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.613768 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:15.706187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.704851 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" event={"ID":"0e717c42-92b7-4a4c-85d7-63574ad07a8b","Type":"ContainerStarted","Data":"dd4f7a4f96aa04243facf766218d8952a4aff3c4b38313b647b491a04c48ccd8"} Nov 02 21:56:15.706187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.704912 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" event={"ID":"0e717c42-92b7-4a4c-85d7-63574ad07a8b","Type":"ContainerStarted","Data":"507ef63084e6d895638c4abf5dc377ca9510a4d0a0763fa0b6cb2a4662bef923"} Nov 02 21:56:15.891458 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:15.891389 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:16.070232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.068934 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:16.358566 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:56:16.358476 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:56:16.386686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.386648 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:16.386942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.386909 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:16.417147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.413636 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:16.468402 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.460209 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:16.468402 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.460272 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:16.598577 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.598428 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:16.598577 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.598490 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:16.610205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.608216 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:16.628328 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.624092 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:16.641059 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.640973 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:16.667401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.667369 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:16.758848 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.758779 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:16.778593 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.778518 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:56:16.811187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.791164 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podStartSLOduration=3.791146082 podStartE2EDuration="3.791146082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:15.834294279 +0000 UTC m=+1314.795844600" watchObservedRunningTime="2026-11-02 21:56:16.791146082 +0000 UTC m=+1315.752696397" Nov 02 21:56:16.963163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:16.958236 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:17.436294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.435829 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:17.450221 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.450171 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:17.468763 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.468729 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:17.468953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.468794 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:17.491168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.481661 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:17.491168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.481723 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:17.619455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.604398 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:17.619455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.604465 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:17.629498 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.629453 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:56:17.630079 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.630056 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:56:17.830690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:17.830653 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:18.132146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.132041 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:18.420489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.420396 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:18.420766 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.420734 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:18.451467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.450621 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:18.451467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.450688 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:18.626963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.626218 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:18.626963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.626279 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:18.628047 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.627318 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:56:18.884148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.883384 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:18.950381 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.950340 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:18.963806 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:18.963709 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:19.139226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.138300 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:19.261150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.241508 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:19.329153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.325165 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:56:19.421226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.415604 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:19.421226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.415658 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:19.421226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.416425 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:19.421226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.417810 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:19.421226 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.417849 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:19.505285 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.505242 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:19.569147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.563723 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:19.670796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.670761 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:19.671342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.671264 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:19.702416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.702312 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:19.893356 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.893299 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:19.896586 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.896561 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:56:19.991150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.984202 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:19.991150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:19.984584 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:56:20.005721 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.005676 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:20.226991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.210980 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:20.283728 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.269495 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:20.283728 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.281674 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:56:20.408258 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.405663 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:20.408541 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.408507 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:20.451147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.407482 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:20.451455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.451412 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:20.452002 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.451976 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:56:20.528148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.488891 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:20.582440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.536689 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:20.582440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.547084 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:56:20.582440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.580034 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:20.610242 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.610205 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:56:20.635295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.611472 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:20.635586 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.635554 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:20.778281 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:20.778245 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz"] Nov 02 21:56:21.288151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.282831 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:21.434151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.361721 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:21.434151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.361779 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz"] Nov 02 21:56:21.434151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.363328 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:21.434151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.363674 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:56:21.434151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.397322 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:21.534925 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.534873 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:21.850631 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.598553 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:56:21.864170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.674581 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:21.864170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.863840 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:21.864170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.850595 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:21.864170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.863927 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:21.870279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.870221 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:21.882148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.877571 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:21.988545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.950416 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8c2719c1-dockercfg-n5ngm\"" Nov 02 21:56:21.997158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:21.951695 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 02 21:56:22.322111 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.088864 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:22.322773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.322738 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:22.322923 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.089805 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8c2719c1\"" Nov 02 21:56:22.395654 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.395604 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.396559 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.396529 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:22.397296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.397270 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.422184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.414994 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.461447 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.456684 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:22.461447 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.456746 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:22.483297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.483258 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:22.483557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.483527 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:22.505809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.505764 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" podUID="cc6364ff-f21b-462b-a93e-f71e85c3e0f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.651210 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.651051 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.709154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.706931 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:22.709154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.706991 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:22.755270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.755212 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrpn\" (UniqueName: \"kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn\") pod \"payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:22.784578 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.755839 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume\") pod \"payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:22.847151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.827401 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.854213 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.851235 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.895274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.895202 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:22.928211 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.913744 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume\") pod \"payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:23.008232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.991383 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume\") pod \"payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:23.008232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:22.991542 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:56:23.159961 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.159907 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:23.187817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.187719 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrpn\" (UniqueName: \"kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn\") pod \"payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:23.267178 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.259349 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:23.333180 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:56:23.323237 4285 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.961s" Nov 02 21:56:23.333180 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.323380 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:56:23.393847 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.393808 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:23.483744 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.483701 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:23.483995 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.446297 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:23.494330 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.416809 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:23.494620 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.442446 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:23.494786 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.494758 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:23.499190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.446327 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:23.510350 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.448758 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrpn\" (UniqueName: \"kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn\") pod \"payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:23.609301 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.598432 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:23.609583 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.609550 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:23.725670 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.725622 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:23.889913 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.889860 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:23.902638 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:23.902603 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:56:24.024164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.024112 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:56:24.024447 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.024430 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv"] Nov 02 21:56:24.036097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036074 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:56:24.036335 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036320 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:56:24.036458 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036448 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:56:24.036556 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036546 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv"] Nov 02 21:56:24.036685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036675 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:56:24.036788 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036779 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:56:24.036894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036885 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:56:24.036999 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.036990 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:56:24.037815 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.037796 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.105807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.105777 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:24.145488 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.145457 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74447\" (UniqueName: \"kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447\") pod \"payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.146633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.146606 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.252560 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.252525 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74447\" (UniqueName: \"kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447\") pod \"payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.253349 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.253320 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.264865 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.260173 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.302155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.292089 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:24.374204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.367759 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448"] Nov 02 21:56:24.374204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.367984 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" containerID="cri-o://28c01a98052af08a2bf7d994021ef4e2ed06853dff906dee345630c4c54dff99" gracePeriod=30 Nov 02 21:56:24.386580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.375249 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:24.386580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.375317 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:24.430797 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.430764 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz"] Nov 02 21:56:24.439194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.436140 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74447\" (UniqueName: \"kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447\") pod \"payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:24.439194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.436239 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:56:24.439194 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.436281 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 21:56:24.520891 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.520857 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:24.598658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.598629 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:24.598800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.598768 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:24.709155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:24.709102 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:25.164688 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.164636 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" event={"ID":"b3ff40c8-e81c-4936-a82f-80a3382e014c","Type":"ContainerStarted","Data":"80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24"} Nov 02 21:56:25.165176 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.165145 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" event={"ID":"b3ff40c8-e81c-4936-a82f-80a3382e014c","Type":"ContainerStarted","Data":"cff870102d9fb9ff645f53041952dbdfefb93604b4fb4980983adfd6f107d21c"} Nov 02 21:56:25.166639 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.166618 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:25.366951 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.366917 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:56:25.367324 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.367292 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 21:56:25.427387 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.427308 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv"] Nov 02 21:56:25.464954 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.464924 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:25.595401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.595345 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" podStartSLOduration=5.595324783 podStartE2EDuration="5.595324783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:25.561764322 +0000 UTC m=+1324.523314643" watchObservedRunningTime="2026-11-02 21:56:25.595324783 +0000 UTC m=+1324.556875103" Nov 02 21:56:25.613030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.606401 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:56:25.613030 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.606459 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 21:56:25.955148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:25.950962 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:26.100545 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.100514 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:26.101784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.101727 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:26.103372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.103313 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 21:56:26.110051 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.109176 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:26.297205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.289193 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" event={"ID":"fd745143-f3cd-43b7-b843-75c5f74c91ae","Type":"ContainerStarted","Data":"02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b"} Nov 02 21:56:26.297205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.290330 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" event={"ID":"fd745143-f3cd-43b7-b843-75c5f74c91ae","Type":"ContainerStarted","Data":"3621dc32fb817e2890e6a780daf49fc0c9151cacb6356ee51c90df823b016242"} Nov 02 21:56:26.297205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.290532 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:26.561199 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.549583 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" podStartSLOduration=6.549566732 podStartE2EDuration="6.549566732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:26.549453552 +0000 UTC m=+1325.511003872" watchObservedRunningTime="2026-11-02 21:56:26.549566732 +0000 UTC m=+1325.511117055" Nov 02 21:56:26.975148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.975044 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:26.980953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.980926 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:26.991454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:26.991427 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 21:56:27.390459 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:27.390382 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:27.392103 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:27.392082 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:27.424254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:27.424227 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 21:56:27.974548 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:27.974514 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9"] Nov 02 21:56:28.152465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:27.985867 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.192695 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.192844 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.192929 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.193052 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.193094 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssdb\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.193219 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.193292 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.193326 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.204153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.193383 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.326536 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.326494 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.326879 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.326837 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.327023 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.327007 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.327195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.327178 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.353186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.329114 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.354507 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.354469 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.354758 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.354741 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.354981 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.354968 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.355097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.355083 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssdb\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.356267 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.356243 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.484666 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.455992 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.484894 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.456262 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.485015 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.456490 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.491184 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.489767 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.500147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.497841 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9"] Nov 02 21:56:28.524541 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.503388 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.524541 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.503467 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:28.558277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.551857 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssdb\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.559678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.559003 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.843206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.834186 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:28.894174 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.893231 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:56:28.897425 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.897387 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:28.953190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.915395 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:28.982527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:28.982481 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.025667 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.025629 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:29.140265 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.135487 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.398227 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.398103 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.414472 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.414073 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:29.530159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.507247 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.530832 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.518783 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.620772 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.620737 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:56:29.914978 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.906526 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:29.914978 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.907474 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:56:29.914978 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.908383 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:56:29.914978 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.908806 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:56:29.995170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:29.995069 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:56:30.194250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.160422 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.312440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.310868 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:56:30.345571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.345531 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:56:30.418306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.418254 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:30.419551 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.419528 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:56:30.439650 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.432595 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.529383 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.529293 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:56:30.687320 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.685350 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6" podUID="cc6364ff-f21b-462b-a93e-f71e85c3e0f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.687320 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.685814 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tw887" podUID="1f8b5643-8dd6-45ee-ba29-6d6d573eaad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.717568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.717525 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:30.735601 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.735255 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:56:30.740144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.735312 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:56:30.880208 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.855101 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:56:30.962016 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.961962 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:30.971797 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.971742 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:30.980614 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:30.972193 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.021148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.006457 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.361352 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.341003 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.517164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.508253 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:31.686975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.661447 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-nw448 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:56:31.686975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.661509 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 21:56:31.686975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.665610 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.686975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.683636 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 02 21:56:31.741811 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.741761 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:31.766762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.766719 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 02 21:56:31.915279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:31.915237 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.010152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.006048 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.151558 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.073969 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:56:32.254959 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.074008 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:56:32.276395 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.151244 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:56:32.279222 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.256369 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 02 21:56:32.284146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.283613 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:56:32.284146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.283682 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.350640 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.350594 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.674246 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.516075 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:32.674246 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.529646 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.711192 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.690238 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:32.749739 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:56:32.738242 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303e2c16_f572_4d46_bb24_57ca0d3b83fa.slice/crio-00596ffaffa642817159101ece57c452ccfe4871362e9e4822864a4befc826a2 WatchSource:0}: Error finding container 00596ffaffa642817159101ece57c452ccfe4871362e9e4822864a4befc826a2: Status 404 returned error can't find the container with id 00596ffaffa642817159101ece57c452ccfe4871362e9e4822864a4befc826a2 Nov 02 21:56:32.816235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.779787 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:32.816235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.782732 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:56:32.926790 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.926746 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:32.972859 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.972804 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:32.973210 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.973168 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:32.973396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:32.973369 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:56:33.070150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.044863 4285 generic.go:356] "Generic (PLEG): container finished" podID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerID="28c01a98052af08a2bf7d994021ef4e2ed06853dff906dee345630c4c54dff99" exitCode=0 Nov 02 21:56:33.164152 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:56:33.150004 4285 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.94s" Nov 02 21:56:33.203211 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.172019 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:56:33.268151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.264493 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.285149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.281950 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.375538 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.369990 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.721260 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.721218 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:33.727149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.726285 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:56:33.727149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.726343 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.727375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.727345 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:33.783608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:33.783567 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:34.177382 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.166338 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:34.270581 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.249351 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:34.319496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.312287 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:34.319496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.313709 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:34.319496 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:56:34.316046 4285 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.134s" Nov 02 21:56:34.319496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.316196 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" event={"ID":"2722e3ac-1614-4185-a842-d6b27f4c86e1","Type":"ContainerDied","Data":"28c01a98052af08a2bf7d994021ef4e2ed06853dff906dee345630c4c54dff99"} Nov 02 21:56:34.466496 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.466452 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:56:34.686147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.663544 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:34.775247 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.752882 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss"] Nov 02 21:56:34.903306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:34.903260 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:56:35.054076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.053985 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:35.080173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.078995 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:35.102804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.099183 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:35.102804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.100409 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 02 21:56:35.270149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.168387 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:35.270149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.261324 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:35.328893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.328860 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:35.532685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.532627 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:56:35.649435 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.649348 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:35.737148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.693957 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:56:35.788152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.742825 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 02 21:56:35.803528 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.803496 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8c2719c1-serving-cert\"" Nov 02 21:56:35.924231 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:35.923405 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:56:36.109725 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.091978 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:36.120347 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.120290 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:36.322374 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.322312 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:36.463929 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.463889 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528296 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528453 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528502 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528561 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528620 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528715 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528812 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.528865 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket\") pod \"2722e3ac-1614-4185-a842-d6b27f4c86e1\" (UID: \"2722e3ac-1614-4185-a842-d6b27f4c86e1\") " Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.529914 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg7s\" (UniqueName: \"kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.529980 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.530077 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:36.531156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.530257 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:36.644567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.527015 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:36.857873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.558086 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:56:36.858138 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.858096 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:36.858296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.849472 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp" podUID="615e6adb-9482-405a-85f3-98d3af5df4cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:36.859081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.857759 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:36.864299 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.864254 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:56:36.891446 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.891399 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:36.892162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.892087 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:56:36.942588 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.942535 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:36.975747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.975701 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:36.976237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:36.976184 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:37.069144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.068631 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg7s\" (UniqueName: \"kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.069144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.068705 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.069376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.069145 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.069376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.069171 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.069376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.069184 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2722e3ac-1614-4185-a842-d6b27f4c86e1-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.069376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.069204 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.069376 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.069219 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.101216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.073788 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:56:37.101216 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.075083 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "kube-api-access-q7fwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:37.158518 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:56:37.152765 4285 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.677s" Nov 02 21:56:37.160484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.158755 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss"] Nov 02 21:56:37.164484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.164457 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9"] Nov 02 21:56:37.249608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.249502 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:56:37.403323 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.251675 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:56:37.403975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.279599 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:56:37.404239 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.279647 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:56:37.404385 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.202882 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:56:37.425552 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.425513 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:56:37.741503 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.741464 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.741686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.741630 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.744576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.744548 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.799989 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.799942 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7fwp\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-kube-api-access-q7fwp\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.801607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.800245 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2722e3ac-1614-4185-a842-d6b27f4c86e1-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.801607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.800272 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.848364 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.848331 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" event={"ID":"2722e3ac-1614-4185-a842-d6b27f4c86e1","Type":"ContainerDied","Data":"89dcd6ee600e670226cb8559637cc0868be1faa54aaf6ff7bf9bd01260ee0558"} Nov 02 21:56:37.848629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.848608 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:56:37.848804 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.848791 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:56:37.848906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.848893 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" event={"ID":"303e2c16-f572-4d46-bb24-57ca0d3b83fa","Type":"ContainerStarted","Data":"00596ffaffa642817159101ece57c452ccfe4871362e9e4822864a4befc826a2"} Nov 02 21:56:37.849857 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.849834 4285 scope.go:122] "RemoveContainer" containerID="28c01a98052af08a2bf7d994021ef4e2ed06853dff906dee345630c4c54dff99" Nov 02 21:56:37.854092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.854062 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:56:37.866316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.865510 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:56:37.866316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.865568 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:56:37.866316 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.865600 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:56:37.879724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.863652 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo" pod "2722e3ac-1614-4185-a842-d6b27f4c86e1" (UID: "2722e3ac-1614-4185-a842-d6b27f4c86e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:56:37.879724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.864203 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.879724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.865436 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.887182 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.886989 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:56:37.898733 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.898707 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg7s\" (UniqueName: \"kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s\") pod \"maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:37.908086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.908035 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2722e3ac-1614-4185-a842-d6b27f4c86e1-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:56:37.910152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.909940 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:56:37.911270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.911246 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:56:37.913363 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.913346 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:56:37.952539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:37.942141 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:38.060408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.060367 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" event={"ID":"303e2c16-f572-4d46-bb24-57ca0d3b83fa","Type":"ContainerStarted","Data":"0d4597c80f5a9090fdb5fe8a966248baaff5e06bf342d8a313b478d7247608f3"} Nov 02 21:56:38.062304 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.062284 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448" Nov 02 21:56:38.460183 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.459558 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podStartSLOduration=11.459537824 podStartE2EDuration="11.459537824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:38.41134847 +0000 UTC m=+1337.372898791" watchObservedRunningTime="2026-11-02 21:56:38.459537824 +0000 UTC m=+1337.421088143" Nov 02 21:56:38.537218 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.537173 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss"] Nov 02 21:56:38.582791 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:56:38.582761 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cad5a3_c086_466c_aa37_0e78fd43e9f3.slice/crio-4e8e058dc9fed1eae0be5c1bee4833506089f00db72f57e33f9dd5dc7634e132 WatchSource:0}: Error finding container 4e8e058dc9fed1eae0be5c1bee4833506089f00db72f57e33f9dd5dc7634e132: Status 404 returned error can't find the container with id 4e8e058dc9fed1eae0be5c1bee4833506089f00db72f57e33f9dd5dc7634e132 Nov 02 21:56:38.584879 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.584857 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:56:38.698200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.698167 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:38.894657 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:38.894569 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:39.052149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.051085 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:39.052358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.052267 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:56:39.052358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.052319 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 21:56:39.070788 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.070759 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" event={"ID":"a0cad5a3-c086-466c-aa37-0e78fd43e9f3","Type":"ContainerStarted","Data":"4e8e058dc9fed1eae0be5c1bee4833506089f00db72f57e33f9dd5dc7634e132"} Nov 02 21:56:39.139149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.139017 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:39.151188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.145044 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448"] Nov 02 21:56:39.151188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.147983 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-nw448"] Nov 02 21:56:39.180744 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:39.180481 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" path="/var/lib/kubelet/pods/2722e3ac-1614-4185-a842-d6b27f4c86e1/volumes" Nov 02 21:56:40.035947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:40.035509 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:56:40.035947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:40.035568 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 21:56:40.079089 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:40.079005 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" event={"ID":"a0cad5a3-c086-466c-aa37-0e78fd43e9f3","Type":"ContainerStarted","Data":"3a9a565714bf111c91af8dae1bcc70bb615937f6d5929addee3d35ba61166497"} Nov 02 21:56:40.079089 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:40.079076 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:40.191568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:40.191432 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" podStartSLOduration=7.987645435 podStartE2EDuration="9.191419613s" totalImagesPullingTime="1.203774178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:40.190640059 +0000 UTC m=+1339.152190379" watchObservedRunningTime="2026-11-02 21:56:40.191419613 +0000 UTC m=+1339.152969933" Nov 02 21:56:41.044151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:41.030512 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:56:41.044151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:41.030579 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 21:56:41.702240 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:41.702206 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:41.872434 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:41.872388 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:42.038207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:42.028557 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:56:42.038207 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:42.028615 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 21:56:42.156443 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:42.156406 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:43.038150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:43.035463 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:56:43.038150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:43.035525 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 21:56:43.864625 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:43.864583 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:44.020692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:44.020648 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:44.026549 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:44.026521 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:44.026810 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:44.026787 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:56:44.026885 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:44.026851 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 21:56:44.171199 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:44.170612 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:45.261991 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:45.261954 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:45.263333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:45.263311 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 21:56:46.044622 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:56:46.044585 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:56:46.081445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.081413 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:46.096974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.096952 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:56:46.265159 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.263652 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:46.715972 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.715938 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd"] Nov 02 21:56:46.717851 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.717810 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" Nov 02 21:56:46.718011 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.717996 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" Nov 02 21:56:46.718618 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.718596 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2722e3ac-1614-4185-a842-d6b27f4c86e1" containerName="istio-proxy" Nov 02 21:56:46.795149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.793510 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:46.844200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.844162 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd"] Nov 02 21:56:46.884147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.868435 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlpc\" (UniqueName: \"kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc\") pod \"payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:46.884567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.884534 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume\") pod \"payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:46.931617 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.931583 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-00052a2d-dockercfg-xkffv\"" Nov 02 21:56:46.938102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.932564 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-00052a2d\"" Nov 02 21:56:47.005254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:46.946866 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww"] Nov 02 21:56:47.015423 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.006996 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlpc\" (UniqueName: \"kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc\") pod \"payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:47.015423 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.007143 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume\") pod \"payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:47.015423 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.008334 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume\") pod \"payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:47.036808 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.020786 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww"] Nov 02 21:56:47.037111 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.028085 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.100206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.080985 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlpc\" (UniqueName: \"kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc\") pod \"payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:47.110347 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.110314 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.110896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.110872 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdfj\" (UniqueName: \"kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj\") pod \"payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.214145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.212208 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdfj\" (UniqueName: \"kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj\") pod \"payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.214145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.212315 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.228156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.214576 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.297390 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.283944 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdfj\" (UniqueName: \"kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj\") pod \"payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.343237 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.337444 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:47.434489 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.434460 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:47.749088 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:56:47.749056 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7d11a4_62ae_4d6f_a999_b45e0e26da9c.slice/crio-d0ac68a7e43d0abeffd9b19e930c882c13d3fe7b67062677e91f0bbba9d615f7 WatchSource:0}: Error finding container d0ac68a7e43d0abeffd9b19e930c882c13d3fe7b67062677e91f0bbba9d615f7: Status 404 returned error can't find the container with id d0ac68a7e43d0abeffd9b19e930c882c13d3fe7b67062677e91f0bbba9d615f7 Nov 02 21:56:47.749711 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.749635 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd"] Nov 02 21:56:47.858985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.858949 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww"] Nov 02 21:56:47.859903 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:56:47.859877 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b62c81_931b_4d92_81e6_1631bf0b3e19.slice/crio-c5722e7b2eb1e0a03d65422f56766dca8ff1a9388f43ad4b575efb3f56b91d19 WatchSource:0}: Error finding container c5722e7b2eb1e0a03d65422f56766dca8ff1a9388f43ad4b575efb3f56b91d19: Status 404 returned error can't find the container with id c5722e7b2eb1e0a03d65422f56766dca8ff1a9388f43ad4b575efb3f56b91d19 Nov 02 21:56:47.998367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:47.998295 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:48.164555 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.164524 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:48.238806 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.238773 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" event={"ID":"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c","Type":"ContainerStarted","Data":"00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a"} Nov 02 21:56:48.238977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.238831 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:48.238977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.238856 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" event={"ID":"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c","Type":"ContainerStarted","Data":"d0ac68a7e43d0abeffd9b19e930c882c13d3fe7b67062677e91f0bbba9d615f7"} Nov 02 21:56:48.240460 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.240435 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" event={"ID":"98b62c81-931b-4d92-81e6-1631bf0b3e19","Type":"ContainerStarted","Data":"2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda"} Nov 02 21:56:48.240579 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.240481 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" event={"ID":"98b62c81-931b-4d92-81e6-1631bf0b3e19","Type":"ContainerStarted","Data":"c5722e7b2eb1e0a03d65422f56766dca8ff1a9388f43ad4b575efb3f56b91d19"} Nov 02 21:56:48.240579 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.240509 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:48.358810 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.358666 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" podStartSLOduration=2.358648489 podStartE2EDuration="2.358648489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:48.351409165 +0000 UTC m=+1347.312959486" watchObservedRunningTime="2026-11-02 21:56:48.358648489 +0000 UTC m=+1347.320198814" Nov 02 21:56:48.529155 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.525841 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" podStartSLOduration=2.525818677 podStartE2EDuration="2.525818677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:48.483453258 +0000 UTC m=+1347.445003579" watchObservedRunningTime="2026-11-02 21:56:48.525818677 +0000 UTC m=+1347.487369001" Nov 02 21:56:48.640519 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:48.640437 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:51.770754 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:51.770716 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:53.415603 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:53.415563 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:57.491829 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:57.491790 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:59.250509 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.250482 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:56:59.250896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.250752 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:56:59.548504 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.548417 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878"] Nov 02 21:56:59.555254 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.555230 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.646119 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.646077 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878"] Nov 02 21:56:59.729440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729408 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8892\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729612 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729476 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729612 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729550 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729614 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729650 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729732 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729818 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729800 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.729919 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.729905 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.730088 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.730074 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.831874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.831801 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.831874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.831839 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8892\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832078 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.831892 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832078 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.831922 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832078 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.831946 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832078 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.831982 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832078 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832018 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832078 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832057 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832441 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832147 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832441 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832239 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832441 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832355 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832560 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832634 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832617 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.832947 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.832923 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.834377 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.834356 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.834516 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.834498 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.888086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.888049 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.888219 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.888205 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8892\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892\") pod \"e2e-worker-w5-96c052-openshift-default-86677f975c-d6878\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:56:59.963996 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:56:59.963967 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:00.171274 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:00.171196 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:57:00.601564 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:57:00.601533 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847e7104_aadd_4903_835f_d71b8dce4beb.slice/crio-7dca9ff0ee7b3bc3a41ece74b18bf75a7788b5461afd0eba64b2cb8b553fc8fe WatchSource:0}: Error finding container 7dca9ff0ee7b3bc3a41ece74b18bf75a7788b5461afd0eba64b2cb8b553fc8fe: Status 404 returned error can't find the container with id 7dca9ff0ee7b3bc3a41ece74b18bf75a7788b5461afd0eba64b2cb8b553fc8fe Nov 02 21:57:00.601913 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:00.601824 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878"] Nov 02 21:57:00.604431 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:00.604403 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:00.604530 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:00.604461 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:00.604530 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:00.604489 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:01.359297 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:01.359260 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" event={"ID":"847e7104-aadd-4903-835f-d71b8dce4beb","Type":"ContainerStarted","Data":"0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a"} Nov 02 21:57:01.359498 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:01.359322 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" event={"ID":"847e7104-aadd-4903-835f-d71b8dce4beb","Type":"ContainerStarted","Data":"7dca9ff0ee7b3bc3a41ece74b18bf75a7788b5461afd0eba64b2cb8b553fc8fe"} Nov 02 21:57:01.528542 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:01.528489 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podStartSLOduration=2.52847547 podStartE2EDuration="2.52847547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:01.525493253 +0000 UTC m=+1360.487043587" watchObservedRunningTime="2026-11-02 21:57:01.52847547 +0000 UTC m=+1360.490025789" Nov 02 21:57:01.886793 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:01.886752 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:02.042147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:02.041663 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:02.172146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:02.172049 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:57:02.173642 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:02.173614 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:57:02.173777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:02.173669 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 21:57:02.990333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:02.990299 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:03.172331 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:03.172295 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:57:03.172605 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:03.172575 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 21:57:04.172477 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:04.172441 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:57:04.172862 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:04.172492 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 21:57:05.171776 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:05.171743 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:57:05.171975 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:05.171802 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 21:57:06.172195 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:06.172114 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:57:06.172594 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:06.172225 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 21:57:07.172521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:07.172472 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:57:07.173051 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:07.172551 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 21:57:07.684344 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:07.684305 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:08.594799 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:08.594765 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:57:08.595352 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:08.595328 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:57:08.596461 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:08.596439 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 21:57:09.893449 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:09.893418 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:12.738462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:12.738414 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.270280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.270233 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.288488 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.288453 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:13.349263 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.349209 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:57:13.547306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.547207 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.669144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.669060 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.741270 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.741221 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:13.742410 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.742381 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:57:13.751467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:13.751431 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:57:14.264666 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.264620 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:57:14.292157 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.278639 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:57:14.293485 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.292431 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:57:14.349791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.349750 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.351926 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.351905 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:57:14.581201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.567477 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.581201 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.570119 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:57:14.599175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.596195 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:57:14.638008 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.637556 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:57:14.638008 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.637627 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.638008 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.637926 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.683415 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.683373 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:57:14.684855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.684830 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:57:14.747511 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.747469 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.767967 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.767927 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:14.768647 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:14.768622 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:57:15.532282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.532227 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:57:15.602160 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.570244 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:57:15.602160 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.571352 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:57:15.623867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.611866 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.675807 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.652871 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.732150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.704234 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:15.732150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.705508 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 02 21:57:15.775152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:15.770335 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:57:16.069238 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.058996 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.137149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.126269 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:57:16.137149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.126341 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:57:16.338362 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.337231 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:57:16.338362 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.337299 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.550229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.533420 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.573154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.572790 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:57:16.573154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.573090 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:57:16.673378 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.673268 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.702902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.699276 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 21:57:16.702902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.699668 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:57:16.735174 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.729479 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:57:16.788568 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:16.788524 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:16.971403 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:57:16.971363 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:57:17.086163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.082411 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:57:17.346329 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.346226 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:57:17.346329 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.346298 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.538225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.537255 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.614208 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.595600 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.661152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.616250 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 21:57:17.703153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.699768 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.703474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.703419 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:17.732242 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.732196 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:57:17.791626 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:17.791578 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:18.099150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.097714 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:57:18.555966 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.551737 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 21:57:18.703867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.703185 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:18.704162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.704113 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:18.714367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.714310 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:18.747062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.720308 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:18.747062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.736822 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 21:57:18.912209 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.887736 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:18.915517 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:18.915476 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:19.107408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.107363 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:57:19.572147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.558890 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:19.708303 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.708249 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:19.708902 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.708870 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 21:57:19.729744 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.729713 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 21:57:19.767146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.741199 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:19.767668 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.756979 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 21:57:19.768032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.757415 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:57:19.903422 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.903309 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 21:57:19.920143 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:19.908494 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 21:57:20.211057 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.209821 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 21:57:20.211057 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.210528 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 21:57:20.238181 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.212017 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 21:57:20.238181 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.219048 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:20.238181 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.234044 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 21:57:20.263610 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.263576 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 21:57:20.270897 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.270871 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x"] Nov 02 21:57:20.278113 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.278090 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 21:57:20.278276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.278120 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x"] Nov 02 21:57:20.278276 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.278226 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 21:57:20.278381 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.278233 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.334149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.332405 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fb2343fe-serving-cert\"" Nov 02 21:57:20.410339 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.410303 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:20.420145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.413984 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.420534 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.420508 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.455977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.453864 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wbt\" (UniqueName: \"kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.455977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.453953 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.601629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.588939 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wbt\" (UniqueName: \"kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.610151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.607520 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.610151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.607658 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.610151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.607732 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.615163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.611592 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.691822 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.691716 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wbt\" (UniqueName: \"kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.693052 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.693027 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.697662 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.696149 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs\") pod \"maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:20.896055 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:20.895969 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:21.337321 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:21.316201 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x"] Nov 02 21:57:22.261053 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:22.261021 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" event={"ID":"6a73eab1-ae93-44bf-b6f7-9c055802d0a5","Type":"ContainerStarted","Data":"df2f09343c58bb0362819bb958af6b852ed6080da9629bcdeed8143622020191"} Nov 02 21:57:23.277365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:23.277166 4285 scope.go:122] "RemoveContainer" containerID="a68d12c3f8926255204db2bcc7043001865b3db33c416a3e750bd56c550a4729" Nov 02 21:57:23.290193 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:23.289728 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" event={"ID":"6a73eab1-ae93-44bf-b6f7-9c055802d0a5","Type":"ContainerStarted","Data":"32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412"} Nov 02 21:57:23.290867 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:23.290843 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:23.407513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:23.407464 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" podStartSLOduration=3.5452540299999997 podStartE2EDuration="4.407450931s" totalImagesPullingTime="862.196901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:23.407047167 +0000 UTC m=+1382.368597488" watchObservedRunningTime="2026-11-02 21:57:23.407450931 +0000 UTC m=+1382.369001251" Nov 02 21:57:25.200567 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:25.200526 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:28.381962 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:28.381874 4285 generic.go:356] "Generic (PLEG): container finished" podID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerID="dd4f7a4f96aa04243facf766218d8952a4aff3c4b38313b647b491a04c48ccd8" exitCode=0 Nov 02 21:57:28.382373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:28.381950 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" event={"ID":"0e717c42-92b7-4a4c-85d7-63574ad07a8b","Type":"ContainerDied","Data":"dd4f7a4f96aa04243facf766218d8952a4aff3c4b38313b647b491a04c48ccd8"} Nov 02 21:57:28.382491 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:28.382476 4285 scope.go:122] "RemoveContainer" containerID="dd4f7a4f96aa04243facf766218d8952a4aff3c4b38313b647b491a04c48ccd8" Nov 02 21:57:28.384083 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:28.384053 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:28.384162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:28.384121 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:28.384203 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:28.384177 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:29.392049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.392016 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" event={"ID":"0e717c42-92b7-4a4c-85d7-63574ad07a8b","Type":"ContainerStarted","Data":"4a4f39299688a8cffd3e50f23b918a9372c876a813397d0cfb598b715c9fa8a2"} Nov 02 21:57:29.620158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.620007 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:57:29.620372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.620169 4285 patch_prober.go:29] interesting pod/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:57:29.620372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.620211 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 21:57:29.756086 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.756044 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6"] Nov 02 21:57:29.763453 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.763425 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.792572 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.792539 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7nf\" (UniqueName: \"kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.792730 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.792651 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.792827 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.792813 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.792898 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.792886 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.793062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.793039 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6"] Nov 02 21:57:29.818148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.816774 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-00052a2d-serving-cert\"" Nov 02 21:57:29.894264 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.894227 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.895250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.895220 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7nf\" (UniqueName: \"kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.895377 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.895316 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.895562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.895537 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.895817 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.895795 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.895922 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:57:29.895906 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-serving-cert: secret "maas-api-e2e-mt-00052a2d-serving-cert" not found Nov 02 21:57:29.895987 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:57:29.895959 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls podName:2ca2dfe2-e650-4c8b-bd92-297b51d5973a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:30.395945586 +0000 UTC m=+1389.357495883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls") pod "maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" (UID: "2ca2dfe2-e650-4c8b-bd92-297b51d5973a") : secret "maas-api-e2e-mt-00052a2d-serving-cert" not found Nov 02 21:57:29.898539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.898513 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:29.951004 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.950968 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:29.953036 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:29.953008 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7nf\" (UniqueName: \"kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:30.141235 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.136406 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:30.322164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.320657 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:30.414214 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.409777 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:57:30.503167 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.494449 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:30.504307 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.504279 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\") pod \"maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:30.629153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.625638 4285 patch_prober.go:29] interesting pod/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:57:30.629153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.625700 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 21:57:30.689051 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:30.688568 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:31.138413 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:31.138376 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6"] Nov 02 21:57:31.140259 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:57:31.140228 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca2dfe2_e650_4c8b_bd92_297b51d5973a.slice/crio-2b053c7f82dbb21472e325232aa40526429ac9359f06c88c9de9068e81b230ed WatchSource:0}: Error finding container 2b053c7f82dbb21472e325232aa40526429ac9359f06c88c9de9068e81b230ed: Status 404 returned error can't find the container with id 2b053c7f82dbb21472e325232aa40526429ac9359f06c88c9de9068e81b230ed Nov 02 21:57:31.469193 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:31.469154 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" event={"ID":"2ca2dfe2-e650-4c8b-bd92-297b51d5973a","Type":"ContainerStarted","Data":"2b053c7f82dbb21472e325232aa40526429ac9359f06c88c9de9068e81b230ed"} Nov 02 21:57:31.636150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:31.635606 4285 patch_prober.go:29] interesting pod/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:57:31.636150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:31.635669 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 21:57:31.743301 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:31.743218 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:57:32.478828 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:32.478791 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" event={"ID":"2ca2dfe2-e650-4c8b-bd92-297b51d5973a","Type":"ContainerStarted","Data":"67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a"} Nov 02 21:57:32.479300 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:32.478877 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:32.598031 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:32.597980 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" podStartSLOduration=2.771610464 podStartE2EDuration="3.597965037s" totalImagesPullingTime="826.354573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:32.596384074 +0000 UTC m=+1391.557934413" watchObservedRunningTime="2026-11-02 21:57:32.597965037 +0000 UTC m=+1391.559515358" Nov 02 21:57:32.637638 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:32.637598 4285 patch_prober.go:29] interesting pod/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:57:32.637826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:32.637656 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 21:57:33.630204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:33.629533 4285 patch_prober.go:29] interesting pod/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:57:33.630204 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:33.629597 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 21:57:34.637314 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:34.637274 4285 patch_prober.go:29] interesting pod/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:57:34.637936 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:34.637886 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 21:57:34.827187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:34.827152 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:35.622667 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:35.622637 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:57:35.623705 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:35.623681 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:57:38.492260 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:38.492230 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:57:39.664963 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:39.664925 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:43.231770 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.231737 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6"] Nov 02 21:57:43.235083 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.235063 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.307467 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.307430 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6"] Nov 02 21:57:43.376836 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.376797 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377006 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.376842 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhppj\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377006 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.376927 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377101 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.377064 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377181 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.377118 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377284 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.377269 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377389 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.377371 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.377418 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.377524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.377471 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.478638 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.478603 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.478795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.478671 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.478795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.478781 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.478904 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.478881 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.478953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.478920 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.478953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.478947 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479036 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479062 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479044 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479068 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhppj\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479202 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479105 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479522 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479600 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479663 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479644 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.479759 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.479709 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.481451 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.481430 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.481950 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.481893 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.535259 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.535231 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.535881 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.535853 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhppj\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj\") pod \"e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.552777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.552753 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:43.985535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.985504 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6"] Nov 02 21:57:43.986035 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:57:43.986011 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1add337_86c2_4e46_b123_b29ed4f3895f.slice/crio-f43f67cd024443771fcee71648fa2000227ac32d6a2469be3ae60b9c0d04ae70 WatchSource:0}: Error finding container f43f67cd024443771fcee71648fa2000227ac32d6a2469be3ae60b9c0d04ae70: Status 404 returned error can't find the container with id f43f67cd024443771fcee71648fa2000227ac32d6a2469be3ae60b9c0d04ae70 Nov 02 21:57:43.988875 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.988845 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:43.988953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.988908 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:43.988953 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:43.988938 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:44.568205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:44.568170 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" event={"ID":"a1add337-86c2-4e46-b123-b29ed4f3895f","Type":"ContainerStarted","Data":"c3ed6149a880a0e5d42ba592dafe35dfd9610567ad14fc11b8539cccae39001c"} Nov 02 21:57:44.568692 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:44.568233 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" event={"ID":"a1add337-86c2-4e46-b123-b29ed4f3895f","Type":"ContainerStarted","Data":"f43f67cd024443771fcee71648fa2000227ac32d6a2469be3ae60b9c0d04ae70"} Nov 02 21:57:44.604639 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:44.604608 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:44.756149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:44.755262 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podStartSLOduration=1.755241715 podStartE2EDuration="1.755241715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:44.688235639 +0000 UTC m=+1403.649785960" watchObservedRunningTime="2026-11-02 21:57:44.755241715 +0000 UTC m=+1403.716792037" Nov 02 21:57:45.553561 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:45.553527 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:45.555444 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:45.555416 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:57:45.555587 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:45.555469 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 21:57:46.008457 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:57:46.008425 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:57:46.553295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:46.553255 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:57:46.553513 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:46.553320 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 21:57:47.553954 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:47.553911 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:57:47.554380 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:47.553987 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 21:57:48.425153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:48.424910 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:48.554061 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:48.554024 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:57:48.554486 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:48.554090 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 21:57:49.257921 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:49.257887 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:49.553745 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:49.553652 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:57:49.554017 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:49.553974 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 21:57:50.554193 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:50.554156 4285 patch_prober.go:29] interesting pod/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:57:50.554791 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:50.554217 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 21:57:51.558595 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:51.558561 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:51.559082 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:51.558944 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:51.559777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:51.559758 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 21:57:54.415032 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:54.414997 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:54.696691 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:54.696657 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz"] Nov 02 21:57:54.823389 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:54.823355 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv"] Nov 02 21:57:55.077462 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:55.077374 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:55.294574 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:55.294496 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:59.201385 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:59.201351 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:59.697271 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:59.697206 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" podUID="b3ff40c8-e81c-4936-a82f-80a3382e014c" containerName="payload-processing" containerID="cri-o://80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24" gracePeriod=25 Nov 02 21:57:59.824353 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:59.824319 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" podUID="fd745143-f3cd-43b7-b843-75c5f74c91ae" containerName="payload-pre-processing" containerID="cri-o://02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b" gracePeriod=25 Nov 02 21:57:59.989686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:57:59.989658 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:58:00.115365 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.114956 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn\" (UniqueName: \"kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn\") pod \"b3ff40c8-e81c-4936-a82f-80a3382e014c\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " Nov 02 21:58:00.115557 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.115456 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume\") pod \"b3ff40c8-e81c-4936-a82f-80a3382e014c\" (UID: \"b3ff40c8-e81c-4936-a82f-80a3382e014c\") " Nov 02 21:58:00.115870 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.115841 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume" pod "b3ff40c8-e81c-4936-a82f-80a3382e014c" (UID: "b3ff40c8-e81c-4936-a82f-80a3382e014c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:00.116669 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.116645 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff40c8-e81c-4936-a82f-80a3382e014c-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:00.119028 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.119006 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn" pod "b3ff40c8-e81c-4936-a82f-80a3382e014c" (UID: "b3ff40c8-e81c-4936-a82f-80a3382e014c"). InnerVolumeSpecName "kube-api-access-9hrpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:00.120009 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.119989 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:58:00.217684 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.217610 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume\") pod \"fd745143-f3cd-43b7-b843-75c5f74c91ae\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " Nov 02 21:58:00.218190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.217684 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447\" (UniqueName: \"kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447\") pod \"fd745143-f3cd-43b7-b843-75c5f74c91ae\" (UID: \"fd745143-f3cd-43b7-b843-75c5f74c91ae\") " Nov 02 21:58:00.218190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.218039 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume" pod "fd745143-f3cd-43b7-b843-75c5f74c91ae" (UID: "fd745143-f3cd-43b7-b843-75c5f74c91ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:00.218307 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.218216 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hrpn\" (UniqueName: \"kubernetes.io/projected/b3ff40c8-e81c-4936-a82f-80a3382e014c-kube-api-access-9hrpn\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:00.218307 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.218238 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd745143-f3cd-43b7-b843-75c5f74c91ae-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:00.219690 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.219663 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447" pod "fd745143-f3cd-43b7-b843-75c5f74c91ae" (UID: "fd745143-f3cd-43b7-b843-75c5f74c91ae"). InnerVolumeSpecName "kube-api-access-74447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:00.320045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.319871 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74447\" (UniqueName: \"kubernetes.io/projected/fd745143-f3cd-43b7-b843-75c5f74c91ae-kube-api-access-74447\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:00.748761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.748727 4285 generic.go:356] "Generic (PLEG): container finished" podID="b3ff40c8-e81c-4936-a82f-80a3382e014c" containerID="80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24" exitCode=0 Nov 02 21:58:00.748914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.748770 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" Nov 02 21:58:00.748914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.748811 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" event={"ID":"b3ff40c8-e81c-4936-a82f-80a3382e014c","Type":"ContainerDied","Data":"80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24"} Nov 02 21:58:00.748914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.748862 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz" event={"ID":"b3ff40c8-e81c-4936-a82f-80a3382e014c","Type":"ContainerDied","Data":"cff870102d9fb9ff645f53041952dbdfefb93604b4fb4980983adfd6f107d21c"} Nov 02 21:58:00.748914 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.748891 4285 scope.go:122] "RemoveContainer" containerID="80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24" Nov 02 21:58:00.750547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.750225 4285 generic.go:356] "Generic (PLEG): container finished" podID="fd745143-f3cd-43b7-b843-75c5f74c91ae" containerID="02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b" exitCode=0 Nov 02 21:58:00.750547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.750282 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" Nov 02 21:58:00.750547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.750296 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" event={"ID":"fd745143-f3cd-43b7-b843-75c5f74c91ae","Type":"ContainerDied","Data":"02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b"} Nov 02 21:58:00.750547 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.750341 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv" event={"ID":"fd745143-f3cd-43b7-b843-75c5f74c91ae","Type":"ContainerDied","Data":"3621dc32fb817e2890e6a780daf49fc0c9151cacb6356ee51c90df823b016242"} Nov 02 21:58:00.760648 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.760630 4285 scope.go:122] "RemoveContainer" containerID="80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24" Nov 02 21:58:00.760914 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:58:00.760895 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24\": container with ID starting with 80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24 not found: ID does not exist" containerID="80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24" Nov 02 21:58:00.760974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.760921 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24"} err="failed to get container status \"80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24\": rpc error: code = NotFound desc = could not find container \"80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24\": container with ID starting with 80380d0f3157c6d77015916010e65a35239f6b29dac1402445091c6706a3ba24 not found: ID does not exist" Nov 02 21:58:00.760974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.760939 4285 scope.go:122] "RemoveContainer" containerID="02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b" Nov 02 21:58:00.770510 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.770493 4285 scope.go:122] "RemoveContainer" containerID="02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b" Nov 02 21:58:00.770747 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:58:00.770731 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b\": container with ID starting with 02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b not found: ID does not exist" containerID="02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b" Nov 02 21:58:00.770795 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.770753 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b"} err="failed to get container status \"02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b\": rpc error: code = NotFound desc = could not find container \"02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b\": container with ID starting with 02ea08556331b3958ad0b0b0415a84b87de157ed7256195520d741830b88da2b not found: ID does not exist" Nov 02 21:58:00.915742 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.915710 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz"] Nov 02 21:58:00.919560 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:00.919536 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c2719c1-74f46cf995-hxdcz"] Nov 02 21:58:01.074633 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:01.074561 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv"] Nov 02 21:58:01.078360 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:01.078297 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c2719c1-7c5874f6cf-c44lv"] Nov 02 21:58:01.175845 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:01.175814 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ff40c8-e81c-4936-a82f-80a3382e014c" path="/var/lib/kubelet/pods/b3ff40c8-e81c-4936-a82f-80a3382e014c/volumes" Nov 02 21:58:01.176282 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:01.176268 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd745143-f3cd-43b7-b843-75c5f74c91ae" path="/var/lib/kubelet/pods/fd745143-f3cd-43b7-b843-75c5f74c91ae/volumes" Nov 02 21:58:01.981473 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:01.981441 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:02.141870 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:02.137883 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:04.081426 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:04.081382 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:08.900498 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:08.900456 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:10.231200 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:10.231161 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:10.398382 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:10.398350 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:13.605930 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:13.605896 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:16.006583 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:58:16.006550 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:58:18.247761 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:18.247726 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:21.922929 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:21.922870 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:22.083465 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:22.083433 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:23.033020 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:23.032985 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:27.936452 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:27.936414 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:31.942191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:31.940261 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss"] Nov 02 21:58:31.942191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:31.940625 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" podUID="a0cad5a3-c086-466c-aa37-0e78fd43e9f3" containerName="maas-api" containerID="cri-o://3a9a565714bf111c91af8dae1bcc70bb615937f6d5929addee3d35ba61166497" gracePeriod=30 Nov 02 21:58:32.128908 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.125069 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:32.136326 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.135847 4285 generic.go:356] "Generic (PLEG): container finished" podID="a0cad5a3-c086-466c-aa37-0e78fd43e9f3" containerID="3a9a565714bf111c91af8dae1bcc70bb615937f6d5929addee3d35ba61166497" exitCode=0 Nov 02 21:58:32.136678 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.135879 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" event={"ID":"a0cad5a3-c086-466c-aa37-0e78fd43e9f3","Type":"ContainerDied","Data":"3a9a565714bf111c91af8dae1bcc70bb615937f6d5929addee3d35ba61166497"} Nov 02 21:58:32.771043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.771016 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:58:32.873232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.868891 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s\" (UniqueName: \"kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s\") pod \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " Nov 02 21:58:32.873232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.869085 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls\") pod \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " Nov 02 21:58:32.873232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.869320 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs\") pod \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " Nov 02 21:58:32.873232 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.869425 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca\") pod \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\" (UID: \"a0cad5a3-c086-466c-aa37-0e78fd43e9f3\") " Nov 02 21:58:32.918097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.905979 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca" pod "a0cad5a3-c086-466c-aa37-0e78fd43e9f3" (UID: "a0cad5a3-c086-466c-aa37-0e78fd43e9f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:32.950147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.943397 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs" pod "a0cad5a3-c086-466c-aa37-0e78fd43e9f3" (UID: "a0cad5a3-c086-466c-aa37-0e78fd43e9f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:32.950147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.943545 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls" pod "a0cad5a3-c086-466c-aa37-0e78fd43e9f3" (UID: "a0cad5a3-c086-466c-aa37-0e78fd43e9f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:32.977256 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:32.967795 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s" pod "a0cad5a3-c086-466c-aa37-0e78fd43e9f3" (UID: "a0cad5a3-c086-466c-aa37-0e78fd43e9f3"). InnerVolumeSpecName "kube-api-access-5gg7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:33.011158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.002871 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:33.011158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.002913 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gg7s\" (UniqueName: \"kubernetes.io/projected/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-kube-api-access-5gg7s\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:33.011158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.002930 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:33.011158 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.002944 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0cad5a3-c086-466c-aa37-0e78fd43e9f3-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:33.163628 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.163588 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" event={"ID":"a0cad5a3-c086-466c-aa37-0e78fd43e9f3","Type":"ContainerDied","Data":"4e8e058dc9fed1eae0be5c1bee4833506089f00db72f57e33f9dd5dc7634e132"} Nov 02 21:58:33.163893 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.163876 4285 scope.go:122] "RemoveContainer" containerID="3a9a565714bf111c91af8dae1bcc70bb615937f6d5929addee3d35ba61166497" Nov 02 21:58:33.164175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.164157 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss" Nov 02 21:58:33.319656 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.319581 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss"] Nov 02 21:58:33.374608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.374559 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c2719c1-79fcdfd749-4qjss"] Nov 02 21:58:33.632816 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.632723 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:33.906097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:33.906024 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:34.087081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:34.086627 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:34.439373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:34.439337 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:34.753146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:34.749068 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:34.907151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:34.906886 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:35.070595 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.070506 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:35.176311 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.176281 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0cad5a3-c086-466c-aa37-0e78fd43e9f3" path="/var/lib/kubelet/pods/a0cad5a3-c086-466c-aa37-0e78fd43e9f3/volumes" Nov 02 21:58:35.367502 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.367422 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:35.536960 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.536927 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk"] Nov 02 21:58:35.537624 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537609 4285 cpu_manager.go:414] "containerMap: removing container" podUID="fd745143-f3cd-43b7-b843-75c5f74c91ae" containerName="payload-pre-processing" Nov 02 21:58:35.537681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537625 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd745143-f3cd-43b7-b843-75c5f74c91ae" containerName="payload-pre-processing" Nov 02 21:58:35.537681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537664 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b3ff40c8-e81c-4936-a82f-80a3382e014c" containerName="payload-processing" Nov 02 21:58:35.537681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537670 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ff40c8-e81c-4936-a82f-80a3382e014c" containerName="payload-processing" Nov 02 21:58:35.537784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537690 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a0cad5a3-c086-466c-aa37-0e78fd43e9f3" containerName="maas-api" Nov 02 21:58:35.537784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537695 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cad5a3-c086-466c-aa37-0e78fd43e9f3" containerName="maas-api" Nov 02 21:58:35.537784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537780 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cad5a3-c086-466c-aa37-0e78fd43e9f3" containerName="maas-api" Nov 02 21:58:35.537874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537790 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd745143-f3cd-43b7-b843-75c5f74c91ae" containerName="payload-pre-processing" Nov 02 21:58:35.537874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.537802 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ff40c8-e81c-4936-a82f-80a3382e014c" containerName="payload-processing" Nov 02 21:58:35.542603 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.542586 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.607147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.600504 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-34089f5b-openshift-default-dockercfg-hzzd7\"" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.696709 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708334 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708411 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708566 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708597 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708667 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmwr\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708701 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708738 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708806 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.808206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.708838 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813218 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813300 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813448 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813475 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813546 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmwr\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813576 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813612 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813682 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.813714 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.814476 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.814735 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.834149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.818790 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.877708 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.840549 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.877708 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.842011 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.921958 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.918752 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmwr\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.921958 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.921053 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.929145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.923614 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:35.986173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:35.986143 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk"] Nov 02 21:58:36.051356 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.025713 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token\") pod \"e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:36.074082 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.067930 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:36.158295 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.156883 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:36.764770 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:58:36.764736 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb124196_7f89_46f5_a6cc_21782b23a2bc.slice/crio-343e1c5b6c00f34d98881dac4a50636230fe05d189e008482c59b01ec9646ef2 WatchSource:0}: Error finding container 343e1c5b6c00f34d98881dac4a50636230fe05d189e008482c59b01ec9646ef2: Status 404 returned error can't find the container with id 343e1c5b6c00f34d98881dac4a50636230fe05d189e008482c59b01ec9646ef2 Nov 02 21:58:36.771419 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.771386 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk"] Nov 02 21:58:36.776800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.776092 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:36.776800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.776246 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:36.776800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:36.776330 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:37.313585 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:37.313533 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" event={"ID":"db124196-7f89-46f5-a6cc-21782b23a2bc","Type":"ContainerStarted","Data":"30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80"} Nov 02 21:58:37.313777 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:37.313635 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" event={"ID":"db124196-7f89-46f5-a6cc-21782b23a2bc","Type":"ContainerStarted","Data":"343e1c5b6c00f34d98881dac4a50636230fe05d189e008482c59b01ec9646ef2"} Nov 02 21:58:37.314358 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:37.314334 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:38.158296 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:38.158264 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:38.164279 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:38.164254 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:38.279283 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:38.279234 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podStartSLOduration=3.279218131 podStartE2EDuration="3.279218131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:37.440040046 +0000 UTC m=+1456.401590369" watchObservedRunningTime="2026-11-02 21:58:38.279218131 +0000 UTC m=+1457.240768451" Nov 02 21:58:38.322615 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:38.322583 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:38.323809 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:38.323784 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 21:58:39.444856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:39.444812 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:39.604665 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:39.604635 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:39.876111 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:39.876013 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:40.287582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:40.287547 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:40.479927 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:40.479883 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:40.706580 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:40.706547 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:41.118197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:41.114096 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:41.412043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:41.411953 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj"] Nov 02 21:58:41.474345 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:41.474301 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" containerID="cri-o://4a4f39299688a8cffd3e50f23b918a9372c876a813397d0cfb598b715c9fa8a2" gracePeriod=30 Nov 02 21:58:41.510152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:41.498003 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:41.701501 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:41.701464 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:42.151260 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.151119 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269"] Nov 02 21:58:42.156527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.156505 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.198172 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.198119 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.198445 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.198423 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.198661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.198643 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.198904 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.198877 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.200719 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.200674 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.200956 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.200937 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.201186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.201160 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.201377 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.201347 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4rd\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.201600 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.201585 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.233525 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.233490 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a4e90e53-openshift-default-dockercfg-4hk5c\"" Nov 02 21:58:42.338389 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.338038 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.352836 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.352912 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4rd\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.353034 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.353219 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.353250 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.353277 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.353946 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.354025 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.354155 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.354704 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.358196 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.354943 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.359976 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.359045 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.359976 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.359342 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.360345 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.360323 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.364375 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.364351 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.434686 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.423260 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269"] Nov 02 21:58:42.763177 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.752363 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:42.776474 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.776398 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4rd\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.787175 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.783595 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:42.923353 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:42.918783 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token\") pod \"e2e-ait-a4e90e53-openshift-default-854fb84545-xc269\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:43.073503 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.073418 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:43.513788 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.513753 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269"] Nov 02 21:58:43.518112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.518065 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:43.518250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.518219 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:43.518360 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.518338 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:43.650164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.649019 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" event={"ID":"ea7abda0-213a-41bb-be21-4f6916d45cfe","Type":"ContainerStarted","Data":"02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb"} Nov 02 21:58:43.650164 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.649070 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" event={"ID":"ea7abda0-213a-41bb-be21-4f6916d45cfe","Type":"ContainerStarted","Data":"1f2caccfb91f1585e663fa951f787d09993597cb0d9ba6a32887b4a1f64ad4a3"} Nov 02 21:58:43.767190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:43.767035 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podStartSLOduration=1.7670198799999999 podStartE2EDuration="1.76701988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:43.765738576 +0000 UTC m=+1462.727288919" watchObservedRunningTime="2026-11-02 21:58:43.76701988 +0000 UTC m=+1462.728570206" Nov 02 21:58:44.074425 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:44.074341 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:44.075873 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:44.075829 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 21:58:44.075951 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:44.075931 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 21:58:44.190904 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:44.190843 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:44.425810 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:44.425693 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:44.586716 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:44.585793 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:45.346146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:45.338648 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:45.553112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:45.553072 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:45.763769 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:45.763730 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:45.765205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:45.765180 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 21:58:45.805224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:45.805198 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:46.006047 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.005995 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:46.010064 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:58:46.010033 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:58:46.276278 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.276229 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:46.768782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.768748 4285 generic.go:356] "Generic (PLEG): container finished" podID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerID="4a4f39299688a8cffd3e50f23b918a9372c876a813397d0cfb598b715c9fa8a2" exitCode=0 Nov 02 21:58:46.769942 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.769917 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" event={"ID":"0e717c42-92b7-4a4c-85d7-63574ad07a8b","Type":"ContainerDied","Data":"4a4f39299688a8cffd3e50f23b918a9372c876a813397d0cfb598b715c9fa8a2"} Nov 02 21:58:46.770043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.769977 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" event={"ID":"0e717c42-92b7-4a4c-85d7-63574ad07a8b","Type":"ContainerDied","Data":"507ef63084e6d895638c4abf5dc377ca9510a4d0a0763fa0b6cb2a4662bef923"} Nov 02 21:58:46.770043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.770000 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507ef63084e6d895638c4abf5dc377ca9510a4d0a0763fa0b6cb2a4662bef923" Nov 02 21:58:46.770043 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.770018 4285 scope.go:122] "RemoveContainer" containerID="dd4f7a4f96aa04243facf766218d8952a4aff3c4b38313b647b491a04c48ccd8" Nov 02 21:58:46.797425 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.797395 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:58:46.888915 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.888839 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.888915 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.888897 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.888965 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889000 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889032 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889188 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889088 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889215 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:46.889416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889279 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889346 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:46.889416 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889386 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889436 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo\") pod \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\" (UID: \"0e717c42-92b7-4a4c-85d7-63574ad07a8b\") " Nov 02 21:58:46.889608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889544 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:46.889608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889557 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:46.889747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.889607 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:46.890102 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.890084 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.890225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.890105 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.890225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.890119 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.890225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.890145 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.890225 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.890157 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.891556 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.891532 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "kube-api-access-rd89f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:46.891643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.891561 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:46.891643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.891565 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:46.891643 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.891616 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo" pod "0e717c42-92b7-4a4c-85d7-63574ad07a8b" (UID: "0e717c42-92b7-4a4c-85d7-63574ad07a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:46.992277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.992200 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.992277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.992228 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd89f\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-kube-api-access-rd89f\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.992277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.992239 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:46.992277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:46.992248 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e717c42-92b7-4a4c-85d7-63574ad07a8b-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:47.778045 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:47.778015 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj" Nov 02 21:58:48.059934 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:48.059854 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj"] Nov 02 21:58:48.063629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:48.063601 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a16a02cb-gw-openshift-default-6948f94649-mndvj"] Nov 02 21:58:49.174908 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:49.174874 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" path="/var/lib/kubelet/pods/0e717c42-92b7-4a4c-85d7-63574ad07a8b/volumes" Nov 02 21:58:50.601372 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:50.601338 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:50.847599 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:50.845029 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:51.186784 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.186739 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm"] Nov 02 21:58:51.190974 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.188052 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" Nov 02 21:58:51.191333 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.191311 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" Nov 02 21:58:51.191562 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.191548 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" Nov 02 21:58:51.191659 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.191647 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" Nov 02 21:58:51.191960 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.191944 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" Nov 02 21:58:51.212152 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.208230 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.329397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.323692 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a4e90e53-dockercfg-k7tsw\"" Nov 02 21:58:51.329397 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.323779 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a4e90e53\"" Nov 02 21:58:51.440208 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.438845 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxwp\" (UniqueName: \"kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp\") pod \"payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.440208 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.438937 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.544762 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.544735 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm"] Nov 02 21:58:51.544977 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.544966 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj"] Nov 02 21:58:51.547055 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.547033 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e717c42-92b7-4a4c-85d7-63574ad07a8b" containerName="istio-proxy" Nov 02 21:58:51.554360 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.554333 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj"] Nov 02 21:58:51.554724 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.554652 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.618058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.592300 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxwp\" (UniqueName: \"kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp\") pod \"payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.618058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.592372 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.618058 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.593292 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.659959 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.658320 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxwp\" (UniqueName: \"kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp\") pod \"payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.677565 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.677535 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:51.694022 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.693997 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6kr\" (UniqueName: \"kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr\") pod \"payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.694156 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.694071 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.795972 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.795938 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6kr\" (UniqueName: \"kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr\") pod \"payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.796185 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.796048 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.797021 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.796994 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.852280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.852248 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6kr\" (UniqueName: \"kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr\") pod \"payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:51.932250 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:51.932216 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:52.075882 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:52.075850 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm"] Nov 02 21:58:52.076716 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:58:52.076690 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee045427_4e77_46e7_b0e0_9a5cd4adb600.slice/crio-be690efb63098806eea7a2ed6d338b9f45078287682f703523c6d840d1c871d8 WatchSource:0}: Error finding container be690efb63098806eea7a2ed6d338b9f45078287682f703523c6d840d1c871d8: Status 404 returned error can't find the container with id be690efb63098806eea7a2ed6d338b9f45078287682f703523c6d840d1c871d8 Nov 02 21:58:52.309228 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:52.309196 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj"] Nov 02 21:58:52.309339 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:58:52.309194 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0ab3a3_1b20_437a_8225_c58d09bc6a14.slice/crio-8e82424a927aeae12a04e30b36c71890a446bfa8095f8fade0e8e40a52cc95ab WatchSource:0}: Error finding container 8e82424a927aeae12a04e30b36c71890a446bfa8095f8fade0e8e40a52cc95ab: Status 404 returned error can't find the container with id 8e82424a927aeae12a04e30b36c71890a446bfa8095f8fade0e8e40a52cc95ab Nov 02 21:58:53.019157 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.019089 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" event={"ID":"ee045427-4e77-46e7-b0e0-9a5cd4adb600","Type":"ContainerStarted","Data":"fe95b813c2f1d49214de39500cf25699cf36f853fcab485cfa773396b94eb8a8"} Nov 02 21:58:53.019556 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.019175 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" event={"ID":"ee045427-4e77-46e7-b0e0-9a5cd4adb600","Type":"ContainerStarted","Data":"be690efb63098806eea7a2ed6d338b9f45078287682f703523c6d840d1c871d8"} Nov 02 21:58:53.019556 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.019316 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:58:53.020938 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.020916 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" event={"ID":"0a0ab3a3-1b20-437a-8225-c58d09bc6a14","Type":"ContainerStarted","Data":"5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a"} Nov 02 21:58:53.021054 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.020952 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" event={"ID":"0a0ab3a3-1b20-437a-8225-c58d09bc6a14","Type":"ContainerStarted","Data":"8e82424a927aeae12a04e30b36c71890a446bfa8095f8fade0e8e40a52cc95ab"} Nov 02 21:58:53.021054 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.020989 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:58:53.136048 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.136001 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" podStartSLOduration=2.135988882 podStartE2EDuration="2.135988882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:53.130690715 +0000 UTC m=+1472.092241038" watchObservedRunningTime="2026-11-02 21:58:53.135988882 +0000 UTC m=+1472.097539201" Nov 02 21:58:53.264916 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:53.264451 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" podStartSLOduration=2.264429602 podStartE2EDuration="2.264429602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:53.258971825 +0000 UTC m=+1472.220522146" watchObservedRunningTime="2026-11-02 21:58:53.264429602 +0000 UTC m=+1472.225979922" Nov 02 21:58:54.242262 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:54.242223 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd"] Nov 02 21:58:54.344091 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:54.344060 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww"] Nov 02 21:58:54.614408 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:54.614327 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:54.785091 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:54.782896 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:55.820148 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:55.819650 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:59.243238 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.243199 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" podUID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" containerName="payload-processing" containerID="cri-o://00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a" gracePeriod=25 Nov 02 21:58:59.249279 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:58:59.249254 4285 logging.go:55] [core] [Channel #1620 SubChannel #1622]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.87:50052", ServerName: "10.133.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.87:50052: connect: connection refused" Nov 02 21:58:59.344571 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.344533 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" podUID="98b62c81-931b-4d92-81e6-1631bf0b3e19" containerName="payload-pre-processing" containerID="cri-o://2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda" gracePeriod=25 Nov 02 21:58:59.562018 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.561987 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:58:59.634874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.634843 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc\" (UniqueName: \"kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc\") pod \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " Nov 02 21:58:59.635053 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.634888 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume\") pod \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\" (UID: \"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c\") " Nov 02 21:58:59.635406 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.635380 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume" pod "bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" (UID: "bb7d11a4-62ae-4d6f-a999-b45e0e26da9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:59.635693 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.635673 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:59.637454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.637424 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc" pod "bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" (UID: "bb7d11a4-62ae-4d6f-a999-b45e0e26da9c"). InnerVolumeSpecName "kube-api-access-hwlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:59.658003 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.657983 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:58:59.736985 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.736940 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume\") pod \"98b62c81-931b-4d92-81e6-1631bf0b3e19\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " Nov 02 21:58:59.737454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.737431 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj\" (UniqueName: \"kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj\") pod \"98b62c81-931b-4d92-81e6-1631bf0b3e19\" (UID: \"98b62c81-931b-4d92-81e6-1631bf0b3e19\") " Nov 02 21:58:59.738379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.738356 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwlpc\" (UniqueName: \"kubernetes.io/projected/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c-kube-api-access-hwlpc\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:59.740475 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.740453 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume" pod "98b62c81-931b-4d92-81e6-1631bf0b3e19" (UID: "98b62c81-931b-4d92-81e6-1631bf0b3e19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:59.741197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.741177 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj" pod "98b62c81-931b-4d92-81e6-1631bf0b3e19" (UID: "98b62c81-931b-4d92-81e6-1631bf0b3e19"). InnerVolumeSpecName "kube-api-access-mwdfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:59.840230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.840146 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwdfj\" (UniqueName: \"kubernetes.io/projected/98b62c81-931b-4d92-81e6-1631bf0b3e19-kube-api-access-mwdfj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:58:59.840230 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:58:59.840183 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98b62c81-931b-4d92-81e6-1631bf0b3e19-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:00.101629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.101544 4285 generic.go:356] "Generic (PLEG): container finished" podID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" containerID="00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a" exitCode=0 Nov 02 21:59:00.101629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.101588 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" Nov 02 21:59:00.101855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.101629 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" event={"ID":"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c","Type":"ContainerDied","Data":"00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a"} Nov 02 21:59:00.101855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.101679 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" event={"ID":"bb7d11a4-62ae-4d6f-a999-b45e0e26da9c","Type":"ContainerDied","Data":"d0ac68a7e43d0abeffd9b19e930c882c13d3fe7b67062677e91f0bbba9d615f7"} Nov 02 21:59:00.101855 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.101707 4285 scope.go:122] "RemoveContainer" containerID="00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a" Nov 02 21:59:00.103168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.103110 4285 generic.go:356] "Generic (PLEG): container finished" podID="98b62c81-931b-4d92-81e6-1631bf0b3e19" containerID="2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda" exitCode=0 Nov 02 21:59:00.103168 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.103154 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" event={"ID":"98b62c81-931b-4d92-81e6-1631bf0b3e19","Type":"ContainerDied","Data":"2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda"} Nov 02 21:59:00.103331 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.103179 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" Nov 02 21:59:00.103331 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.103203 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww" event={"ID":"98b62c81-931b-4d92-81e6-1631bf0b3e19","Type":"ContainerDied","Data":"c5722e7b2eb1e0a03d65422f56766dca8ff1a9388f43ad4b575efb3f56b91d19"} Nov 02 21:59:00.113455 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.113299 4285 scope.go:122] "RemoveContainer" containerID="00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a" Nov 02 21:59:00.113911 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:59:00.113689 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a\": container with ID starting with 00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a not found: ID does not exist" containerID="00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a" Nov 02 21:59:00.113911 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.113722 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a"} err="failed to get container status \"00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a\": rpc error: code = NotFound desc = could not find container \"00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a\": container with ID starting with 00c6bb87a9dbd12e123583725978db262a016cf1dc0b4db7064a2b0d8b544b2a not found: ID does not exist" Nov 02 21:59:00.113911 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.113743 4285 scope.go:122] "RemoveContainer" containerID="2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda" Nov 02 21:59:00.125165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.125145 4285 scope.go:122] "RemoveContainer" containerID="2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda" Nov 02 21:59:00.125463 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:59:00.125444 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda\": container with ID starting with 2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda not found: ID does not exist" containerID="2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda" Nov 02 21:59:00.125535 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.125475 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda"} err="failed to get container status \"2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda\": rpc error: code = NotFound desc = could not find container \"2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda\": container with ID starting with 2b5e4479bb6ae2b5c449e31089fc8b5c2dd60a6fe949c94d76e8f55e0776cfda not found: ID does not exist" Nov 02 21:59:00.248087 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.248045 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd" podUID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.87:50052\" within 1s: context deadline exceeded" Nov 02 21:59:00.265905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.265878 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd"] Nov 02 21:59:00.315629 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.315605 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-00052a2d-9ddd877f5-jzthd"] Nov 02 21:59:00.422682 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.422604 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww"] Nov 02 21:59:00.426572 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.426541 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-00052a2d-5dd686c56c-bfbww"] Nov 02 21:59:00.836149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:00.835414 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:01.185012 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:01.184934 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b62c81-931b-4d92-81e6-1631bf0b3e19" path="/var/lib/kubelet/pods/98b62c81-931b-4d92-81e6-1631bf0b3e19/volumes" Nov 02 21:59:01.185431 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:01.185415 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" path="/var/lib/kubelet/pods/bb7d11a4-62ae-4d6f-a999-b45e0e26da9c/volumes" Nov 02 21:59:04.031268 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:04.031054 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 21:59:04.031268 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:04.031137 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 21:59:05.501454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:05.501423 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:09.432401 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:09.432365 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x"] Nov 02 21:59:09.432851 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:09.432554 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" podUID="6a73eab1-ae93-44bf-b6f7-9c055802d0a5" containerName="maas-api" containerID="cri-o://32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412" gracePeriod=30 Nov 02 21:59:09.553773 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:09.553742 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:09.975433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:09.975399 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:59:10.093146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.092657 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs\") pod \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " Nov 02 21:59:10.093146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.092906 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt\" (UniqueName: \"kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt\") pod \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " Nov 02 21:59:10.093146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.092992 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls\") pod \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " Nov 02 21:59:10.093146 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.093029 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca\") pod \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\" (UID: \"6a73eab1-ae93-44bf-b6f7-9c055802d0a5\") " Nov 02 21:59:10.111206 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.103034 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca" pod "6a73eab1-ae93-44bf-b6f7-9c055802d0a5" (UID: "6a73eab1-ae93-44bf-b6f7-9c055802d0a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:10.119147 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.113685 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs" pod "6a73eab1-ae93-44bf-b6f7-9c055802d0a5" (UID: "6a73eab1-ae93-44bf-b6f7-9c055802d0a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:10.123197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.120268 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt" pod "6a73eab1-ae93-44bf-b6f7-9c055802d0a5" (UID: "6a73eab1-ae93-44bf-b6f7-9c055802d0a5"). InnerVolumeSpecName "kube-api-access-p6wbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:10.123197 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.120296 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls" pod "6a73eab1-ae93-44bf-b6f7-9c055802d0a5" (UID: "6a73eab1-ae93-44bf-b6f7-9c055802d0a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:10.186266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.186231 4285 generic.go:356] "Generic (PLEG): container finished" podID="6a73eab1-ae93-44bf-b6f7-9c055802d0a5" containerID="32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412" exitCode=0 Nov 02 21:59:10.186454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.186294 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" Nov 02 21:59:10.186454 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.186426 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" event={"ID":"6a73eab1-ae93-44bf-b6f7-9c055802d0a5","Type":"ContainerDied","Data":"32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412"} Nov 02 21:59:10.186576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.186473 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x" event={"ID":"6a73eab1-ae93-44bf-b6f7-9c055802d0a5","Type":"ContainerDied","Data":"df2f09343c58bb0362819bb958af6b852ed6080da9629bcdeed8143622020191"} Nov 02 21:59:10.186576 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.186504 4285 scope.go:122] "RemoveContainer" containerID="32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412" Nov 02 21:59:10.194905 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.194877 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:10.195027 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.194923 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6wbt\" (UniqueName: \"kubernetes.io/projected/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-kube-api-access-p6wbt\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:10.195027 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.194943 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:10.195027 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.194960 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a73eab1-ae93-44bf-b6f7-9c055802d0a5-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:10.201747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.201729 4285 scope.go:122] "RemoveContainer" containerID="32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412" Nov 02 21:59:10.202082 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:59:10.202057 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412\": container with ID starting with 32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412 not found: ID does not exist" containerID="32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412" Nov 02 21:59:10.202191 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.202093 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412"} err="failed to get container status \"32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412\": rpc error: code = NotFound desc = could not find container \"32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412\": container with ID starting with 32813e2b49226cbfa83ce750efc6c62dc216c8f994ca82c1126df3551079d412 not found: ID does not exist" Nov 02 21:59:10.533243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.533200 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x"] Nov 02 21:59:10.600396 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.600362 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb2343fe-86b6bbb87f-btj5x"] Nov 02 21:59:10.605607 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:10.605195 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:11.082856 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:11.082823 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:11.176642 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:11.176610 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a73eab1-ae93-44bf-b6f7-9c055802d0a5" path="/var/lib/kubelet/pods/6a73eab1-ae93-44bf-b6f7-9c055802d0a5/volumes" Nov 02 21:59:11.306658 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:11.306624 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:11.480057 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:11.479932 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:11.679895 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:11.679865 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:12.694581 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:12.694552 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:12.862228 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:12.862193 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:15.693140 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:15.693092 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6"] Nov 02 21:59:15.693581 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:15.693364 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" podUID="2ca2dfe2-e650-4c8b-bd92-297b51d5973a" containerName="maas-api" containerID="cri-o://67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a" gracePeriod=30 Nov 02 21:59:15.845334 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:15.845304 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:16.121145 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:59:16.120405 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:59:16.157243 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.154420 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:59:16.278162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.274108 4285 generic.go:356] "Generic (PLEG): container finished" podID="2ca2dfe2-e650-4c8b-bd92-297b51d5973a" containerID="67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a" exitCode=0 Nov 02 21:59:16.278162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.274259 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" event={"ID":"2ca2dfe2-e650-4c8b-bd92-297b51d5973a","Type":"ContainerDied","Data":"67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a"} Nov 02 21:59:16.278162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.274311 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" event={"ID":"2ca2dfe2-e650-4c8b-bd92-297b51d5973a","Type":"ContainerDied","Data":"2b053c7f82dbb21472e325232aa40526429ac9359f06c88c9de9068e81b230ed"} Nov 02 21:59:16.278162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.274342 4285 scope.go:122] "RemoveContainer" containerID="67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a" Nov 02 21:59:16.278162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.274527 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6" Nov 02 21:59:16.313355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.305766 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf\" (UniqueName: \"kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf\") pod \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " Nov 02 21:59:16.313355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.306609 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\") pod \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " Nov 02 21:59:16.313355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.307244 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs\") pod \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " Nov 02 21:59:16.313355 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.307347 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca\") pod \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\" (UID: \"2ca2dfe2-e650-4c8b-bd92-297b51d5973a\") " Nov 02 21:59:16.342735 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.342701 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca" pod "2ca2dfe2-e650-4c8b-bd92-297b51d5973a" (UID: "2ca2dfe2-e650-4c8b-bd92-297b51d5973a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:16.366162 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.365158 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls" pod "2ca2dfe2-e650-4c8b-bd92-297b51d5973a" (UID: "2ca2dfe2-e650-4c8b-bd92-297b51d5973a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:16.368575 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.368544 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs" pod "2ca2dfe2-e650-4c8b-bd92-297b51d5973a" (UID: "2ca2dfe2-e650-4c8b-bd92-297b51d5973a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:16.429154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.427672 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf" pod "2ca2dfe2-e650-4c8b-bd92-297b51d5973a" (UID: "2ca2dfe2-e650-4c8b-bd92-297b51d5973a"). InnerVolumeSpecName "kube-api-access-nj7nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:16.429154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.428428 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:16.429154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.428457 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:16.429154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.428491 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj7nf\" (UniqueName: \"kubernetes.io/projected/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-kube-api-access-nj7nf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:16.429154 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.428508 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca2dfe2-e650-4c8b-bd92-297b51d5973a-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 21:59:16.469173 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.468317 4285 scope.go:122] "RemoveContainer" containerID="67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a" Nov 02 21:59:16.478750 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:59:16.469668 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a\": container with ID starting with 67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a not found: ID does not exist" containerID="67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a" Nov 02 21:59:16.478750 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.469753 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a"} err="failed to get container status \"67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a\": rpc error: code = NotFound desc = could not find container \"67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a\": container with ID starting with 67fefb5fe5e6d077eb51c83654aafb19db9579ef3148a4c0e2d1fa74df8e688a not found: ID does not exist" Nov 02 21:59:16.758144 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.754778 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6"] Nov 02 21:59:16.795068 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:16.795032 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-00052a2d-577c696d8b-nf8s6"] Nov 02 21:59:17.179887 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:17.179855 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca2dfe2-e650-4c8b-bd92-297b51d5973a" path="/var/lib/kubelet/pods/2ca2dfe2-e650-4c8b-bd92-297b51d5973a/volumes" Nov 02 21:59:17.376651 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:17.376620 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:17.651842 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:17.651809 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:17.801952 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:17.801919 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:18.283187 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:18.283081 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:22.238527 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.238491 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99"] Nov 02 21:59:22.239228 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239213 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2ca2dfe2-e650-4c8b-bd92-297b51d5973a" containerName="maas-api" Nov 02 21:59:22.239280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239229 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca2dfe2-e650-4c8b-bd92-297b51d5973a" containerName="maas-api" Nov 02 21:59:22.239280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239241 4285 cpu_manager.go:414] "containerMap: removing container" podUID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" containerName="payload-processing" Nov 02 21:59:22.239280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239247 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" containerName="payload-processing" Nov 02 21:59:22.239280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239264 4285 cpu_manager.go:414] "containerMap: removing container" podUID="6a73eab1-ae93-44bf-b6f7-9c055802d0a5" containerName="maas-api" Nov 02 21:59:22.239280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239271 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a73eab1-ae93-44bf-b6f7-9c055802d0a5" containerName="maas-api" Nov 02 21:59:22.239433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239307 4285 cpu_manager.go:414] "containerMap: removing container" podUID="98b62c81-931b-4d92-81e6-1631bf0b3e19" containerName="payload-pre-processing" Nov 02 21:59:22.239433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239313 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b62c81-931b-4d92-81e6-1631bf0b3e19" containerName="payload-pre-processing" Nov 02 21:59:22.239433 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239421 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b62c81-931b-4d92-81e6-1631bf0b3e19" containerName="payload-pre-processing" Nov 02 21:59:22.239521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239436 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca2dfe2-e650-4c8b-bd92-297b51d5973a" containerName="maas-api" Nov 02 21:59:22.239521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239444 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7d11a4-62ae-4d6f-a999-b45e0e26da9c" containerName="payload-processing" Nov 02 21:59:22.239521 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.239452 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a73eab1-ae93-44bf-b6f7-9c055802d0a5" containerName="maas-api" Nov 02 21:59:22.244603 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.244583 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.296387 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.296337 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-34089f5b-serving-cert\"" Nov 02 21:59:22.299909 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.299886 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99"] Nov 02 21:59:22.367185 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.367152 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.367373 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.367297 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqzz\" (UniqueName: \"kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.367506 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.367484 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.367573 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.367548 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.449842 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.449810 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:22.469538 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.469501 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.469694 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.469643 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqzz\" (UniqueName: \"kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.469760 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.469731 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.469819 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.469767 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.470479 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.470456 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.472091 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.472069 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.472186 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.472118 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.526476 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.526417 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqzz\" (UniqueName: \"kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz\") pod \"maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.558516 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.558493 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:22.728170 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.706845 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:22.861112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:22.860485 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:23.142566 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:23.142538 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99"] Nov 02 21:59:23.142684 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:59:23.142587 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb54226_a6c6_4cb6_bd5f_98dbba37d05e.slice/crio-15e8fabcbb30f473c14aef886137b12a0143df8516dccff3de55a9435f64b273 WatchSource:0}: Error finding container 15e8fabcbb30f473c14aef886137b12a0143df8516dccff3de55a9435f64b273: Status 404 returned error can't find the container with id 15e8fabcbb30f473c14aef886137b12a0143df8516dccff3de55a9435f64b273 Nov 02 21:59:23.330384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:23.330342 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" event={"ID":"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e","Type":"ContainerStarted","Data":"15e8fabcbb30f473c14aef886137b12a0143df8516dccff3de55a9435f64b273"} Nov 02 21:59:24.087049 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.087015 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:24.288627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.288596 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:24.436627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.436532 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:24.446224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.446176 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" event={"ID":"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e","Type":"ContainerStarted","Data":"633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32"} Nov 02 21:59:24.446664 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.446645 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:24.593151 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.589452 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" podStartSLOduration=1.6800598340000001 podStartE2EDuration="2.589430544s" totalImagesPullingTime="909.37071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:24.570552884 +0000 UTC m=+1503.532103205" watchObservedRunningTime="2026-11-02 21:59:24.589430544 +0000 UTC m=+1503.550980864" Nov 02 21:59:24.908190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:24.907959 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:25.075498 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:25.075427 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:25.389389 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:25.389343 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:27.401685 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:27.401642 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:30.478034 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:30.478001 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 21:59:31.363986 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:31.363947 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:31.585112 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:31.585079 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:32.048388 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:32.048357 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:32.224149 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:32.223634 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:32.472619 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:32.472580 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:36.438095 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:36.438046 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:37.025153 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.003073 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:37.318800 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.318710 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6"] Nov 02 21:59:37.347038 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.347005 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.465280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.465234 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:37.467322 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.467279 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e2be7cc3-openshift-default-dockercfg-82vtr\"" Nov 02 21:59:37.578484 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.578297 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.578874 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.578847 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.579338 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.579313 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.586264 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.582499 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.712796 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.712758 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4w5t\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.713161 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.713065 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.713398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.713364 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.713748 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.713717 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.713889 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.713874 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.714304 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.709963 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:37.861533 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.861437 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.867539 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.867502 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.867976 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.867950 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.868163 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.868147 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.868351 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.868337 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.868464 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.868451 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.868582 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.868568 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4w5t\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.868669 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.868658 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.868782 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.868770 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.887854 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.887818 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.888837 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.888805 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.889550 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.889523 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.890379 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.890358 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.890730 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.890710 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.905367 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.905185 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6"] Nov 02 21:59:37.920161 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.920098 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.928783 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.928752 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.950477 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.950442 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:37.952464 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:37.951703 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4w5t\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:38.034398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.034364 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:38.478440 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.478402 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6"] Nov 02 21:59:38.529190 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.529146 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:38.529310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.529253 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:38.529310 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.529303 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:38.697205 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.697166 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" event={"ID":"65c76022-a683-4360-9d08-24234e95257d","Type":"ContainerStarted","Data":"8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee"} Nov 02 21:59:38.697448 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.697429 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" event={"ID":"65c76022-a683-4360-9d08-24234e95257d","Type":"ContainerStarted","Data":"c793161349528eedb1c808abccce7eb61737d427049d28157fea1aa10e2e3105"} Nov 02 21:59:38.826081 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:38.825960 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podStartSLOduration=1.82594052 podStartE2EDuration="1.82594052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:38.817818301 +0000 UTC m=+1517.779368622" watchObservedRunningTime="2026-11-02 21:59:38.82594052 +0000 UTC m=+1517.787490840" Nov 02 21:59:39.038069 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:39.038016 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 21:59:39.038398 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:39.038337 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 21:59:39.038896 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:39.036018 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:39.659715 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:39.659668 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:40.072150 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:40.058455 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:40.713255 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:40.713209 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:40.714384 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:40.714353 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 21:59:41.651262 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:41.651229 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:41.820911 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:41.820867 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:41.996704 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:41.996656 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:44.801747 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:44.801715 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:45.098160 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.098042 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:45.611280 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.611250 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856"] Nov 02 21:59:45.764094 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.758782 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856"] Nov 02 21:59:45.764094 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.759625 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:45.871731 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.871700 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e2be7cc3\"" Nov 02 21:59:45.875306 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.875253 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e2be7cc3-dockercfg-94gqp\"" Nov 02 21:59:45.882742 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.874341 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:45.891564 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.891537 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvn77\" (UniqueName: \"kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77\") pod \"payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:45.911826 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:45.911789 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.015972 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.015935 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvn77\" (UniqueName: \"kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77\") pod \"payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.016178 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.016092 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.017266 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.017241 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.034439 ip-10-0-131-47 kubenswrapper[4285]: E1102 21:59:46.034407 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 21:59:46.071788 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.071761 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvn77\" (UniqueName: \"kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77\") pod \"payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.217104 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.217072 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.597097 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.597061 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856"] Nov 02 21:59:46.597269 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:59:46.597241 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8d9759_e788_48f0_a6d2_ec9b88271cd1.slice/crio-bc3186eb7436cd1f128c678316a50a77dbbeaf39444e86a68d41c76736ae12ba WatchSource:0}: Error finding container bc3186eb7436cd1f128c678316a50a77dbbeaf39444e86a68d41c76736ae12ba: Status 404 returned error can't find the container with id bc3186eb7436cd1f128c678316a50a77dbbeaf39444e86a68d41c76736ae12ba Nov 02 21:59:46.890871 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.890779 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" event={"ID":"2c8d9759-e788-48f0-a6d2-ec9b88271cd1","Type":"ContainerStarted","Data":"61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76"} Nov 02 21:59:46.890871 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.890846 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:46.890871 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:46.890864 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" event={"ID":"2c8d9759-e788-48f0-a6d2-ec9b88271cd1","Type":"ContainerStarted","Data":"bc3186eb7436cd1f128c678316a50a77dbbeaf39444e86a68d41c76736ae12ba"} Nov 02 21:59:47.005481 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:47.005432 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" podStartSLOduration=2.005416652 podStartE2EDuration="2.005416652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:47.003663653 +0000 UTC m=+1525.965213972" watchObservedRunningTime="2026-11-02 21:59:47.005416652 +0000 UTC m=+1525.966966973" Nov 02 21:59:47.151627 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:47.146338 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:51.793145 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:51.784348 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:54.278679 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.278632 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:54.747354 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.747319 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n"] Nov 02 21:59:54.752520 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.752491 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769013 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.768987 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769024 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769040 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769229 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769092 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv57\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769417 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769251 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769417 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769309 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769524 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769474 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769579 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769530 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.769675 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.769661 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:54.815680 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:54.815641 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-caac72-openshift-default-dockercfg-54rsv\"" Nov 02 21:59:55.061277 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.061183 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.062746 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.062662 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.063224 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.063199 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.063577 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.063546 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.063879 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.063803 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.064076 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.064057 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.064294 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.064276 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv57\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.064480 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.064460 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.064647 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.064630 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.067060 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.066993 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.128608 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.121030 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.129681 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.129628 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.156851 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.154957 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.156851 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.155244 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.158906 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.157627 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.237749 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.237696 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.263342 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.263229 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:55.264212 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.263599 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.282165 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.282086 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n"] Nov 02 21:59:55.357654 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.357572 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv57\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57\") pod \"e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.373869 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.369476 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:55.494171 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.494142 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:55.861810 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.861781 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n"] Nov 02 21:59:55.863304 ip-10-0-131-47 kubenswrapper[4285]: W1102 21:59:55.863278 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a87d989_eba7_430a_ac2c_1c45fe0cef7a.slice/crio-7e269930390c5aad3283df698a7c0ed4a9e0ce98c4a78036f6ec3c1313d3f24a WatchSource:0}: Error finding container 7e269930390c5aad3283df698a7c0ed4a9e0ce98c4a78036f6ec3c1313d3f24a: Status 404 returned error can't find the container with id 7e269930390c5aad3283df698a7c0ed4a9e0ce98c4a78036f6ec3c1313d3f24a Nov 02 21:59:55.866661 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.866625 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:55.866752 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.866700 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:55.866752 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:55.866730 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:59:56.283245 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:56.283201 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" event={"ID":"9a87d989-eba7-430a-ac2c-1c45fe0cef7a","Type":"ContainerStarted","Data":"4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b"} Nov 02 21:59:56.283722 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:56.283267 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" event={"ID":"9a87d989-eba7-430a-ac2c-1c45fe0cef7a","Type":"ContainerStarted","Data":"7e269930390c5aad3283df698a7c0ed4a9e0ce98c4a78036f6ec3c1313d3f24a"} Nov 02 21:59:56.373828 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:56.373796 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:56.380332 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:56.380306 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:56.402540 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:56.402482 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podStartSLOduration=2.40246389 podStartE2EDuration="2.40246389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:56.397310284 +0000 UTC m=+1535.358860605" watchObservedRunningTime="2026-11-02 21:59:56.40246389 +0000 UTC m=+1535.364014211" Nov 02 21:59:56.406464 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:56.406436 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:57.291114 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:57.291082 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:57.292285 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:57.292265 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 21:59:57.903092 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:57.903061 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 21:59:59.741768 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:59.741738 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:59.954442 ip-10-0-131-47 kubenswrapper[4285]: I1102 21:59:59.954411 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:00.122512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.122429 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:00.204873 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.204841 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx"] Nov 02 22:00:00.209570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.209550 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.223650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.223622 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4"] Nov 02 22:00:00.228320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.228302 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.229903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.229885 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv"] Nov 02 22:00:00.234021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.234003 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.263018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.262987 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-7rvc4\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.263165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.263034 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f9c\" (UniqueName: \"kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c\") pod \"maas-api-key-cleanup-29894280-7rvc4\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.263290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.263269 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twr5l\" (UniqueName: \"kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.263349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.263306 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.263425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.263409 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.263476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.263460 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zm4\" (UniqueName: \"kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.270825 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.270808 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:00:00.329450 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.329416 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx"] Nov 02 22:00:00.365422 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.365384 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twr5l\" (UniqueName: \"kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.366021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.365990 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.366248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.366061 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.366248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.366148 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zm4\" (UniqueName: \"kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.366377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.366356 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-7rvc4\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.366431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.366418 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f9c\" (UniqueName: \"kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c\") pod \"maas-api-key-cleanup-29894280-7rvc4\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.368041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.368016 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.368246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.368159 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894280-7rvc4\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.368627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.368591 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.405196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.400811 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4"] Nov 02 22:00:00.409984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.409951 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv"] Nov 02 22:00:00.427962 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.427933 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twr5l\" (UniqueName: \"kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l\") pod \"maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.436402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.436370 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zm4\" (UniqueName: \"kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4\") pod \"maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.436766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.436744 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f9c\" (UniqueName: \"kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c\") pod \"maas-api-key-cleanup-29894280-7rvc4\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.525104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.525065 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:00.542853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.542713 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:00.550652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.550626 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:00.921579 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:00.921547 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7 WatchSource:0}: Error finding container 2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7: Status 404 returned error can't find the container with id 2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7 Nov 02 22:00:00.922189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.922161 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx"] Nov 02 22:00:00.972355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.972332 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4"] Nov 02 22:00:00.972777 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:00.972752 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebed37fd_1beb_4f28_8381_45d6ecc6021b.slice/crio-6e7893533c146dc64b9d2879134ac0efd66e144c79aa914196047fe05da5d396 WatchSource:0}: Error finding container 6e7893533c146dc64b9d2879134ac0efd66e144c79aa914196047fe05da5d396: Status 404 returned error can't find the container with id 6e7893533c146dc64b9d2879134ac0efd66e144c79aa914196047fe05da5d396 Nov 02 22:00:00.976295 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:00.976267 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5751cfdb_44b0_4845_a8fb_e97695942dbc.slice/crio-78a1cd9ccde08620d7a05afaa7677f4656a7140d8c0ee1c13a7bb57e645135c4 WatchSource:0}: Error finding container 78a1cd9ccde08620d7a05afaa7677f4656a7140d8c0ee1c13a7bb57e645135c4: Status 404 returned error can't find the container with id 78a1cd9ccde08620d7a05afaa7677f4656a7140d8c0ee1c13a7bb57e645135c4 Nov 02 22:00:00.978674 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:00.978652 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv"] Nov 02 22:00:01.324989 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:01.324954 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" event={"ID":"ebed37fd-1beb-4f28-8381-45d6ecc6021b","Type":"ContainerStarted","Data":"6e7893533c146dc64b9d2879134ac0efd66e144c79aa914196047fe05da5d396"} Nov 02 22:00:01.326389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:01.326354 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" event={"ID":"5751cfdb-44b0-4845-a8fb-e97695942dbc","Type":"ContainerStarted","Data":"78a1cd9ccde08620d7a05afaa7677f4656a7140d8c0ee1c13a7bb57e645135c4"} Nov 02 22:00:01.328118 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:01.328093 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerStarted","Data":"b380bf8c271f64e227259eb4ca8b719e560820a4cca969cc669e3c784c21d54e"} Nov 02 22:00:01.328239 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:01.328165 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerStarted","Data":"2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7"} Nov 02 22:00:01.446321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:01.446273 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" podStartSLOduration=1.446258177 podStartE2EDuration="1.446258177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:01.440058382 +0000 UTC m=+1540.401608711" watchObservedRunningTime="2026-11-02 22:00:01.446258177 +0000 UTC m=+1540.407808497" Nov 02 22:00:01.663885 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:01.663848 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:02.339043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.339009 4285 generic.go:356] "Generic (PLEG): container finished" podID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerID="21d90a8a5446377618feec2040bfd5433c19bf37da7a3f4d8182c00d4577bbf2" exitCode=60 Nov 02 22:00:02.339547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.339095 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" event={"ID":"ebed37fd-1beb-4f28-8381-45d6ecc6021b","Type":"ContainerDied","Data":"21d90a8a5446377618feec2040bfd5433c19bf37da7a3f4d8182c00d4577bbf2"} Nov 02 22:00:02.339547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.339529 4285 scope.go:122] "RemoveContainer" containerID="21d90a8a5446377618feec2040bfd5433c19bf37da7a3f4d8182c00d4577bbf2" Nov 02 22:00:02.341103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.341081 4285 generic.go:356] "Generic (PLEG): container finished" podID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerID="3bc8cd8cb167c5f916a207f4e8aa2687ce548e106eb74c8e790ebaef3b7ba7f8" exitCode=60 Nov 02 22:00:02.341236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.341211 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" event={"ID":"5751cfdb-44b0-4845-a8fb-e97695942dbc","Type":"ContainerDied","Data":"3bc8cd8cb167c5f916a207f4e8aa2687ce548e106eb74c8e790ebaef3b7ba7f8"} Nov 02 22:00:02.341529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.341510 4285 scope.go:122] "RemoveContainer" containerID="3bc8cd8cb167c5f916a207f4e8aa2687ce548e106eb74c8e790ebaef3b7ba7f8" Nov 02 22:00:02.343294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.343266 4285 generic.go:356] "Generic (PLEG): container finished" podID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerID="b380bf8c271f64e227259eb4ca8b719e560820a4cca969cc669e3c784c21d54e" exitCode=60 Nov 02 22:00:02.343402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.343347 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerDied","Data":"b380bf8c271f64e227259eb4ca8b719e560820a4cca969cc669e3c784c21d54e"} Nov 02 22:00:02.343673 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:02.343572 4285 scope.go:122] "RemoveContainer" containerID="b380bf8c271f64e227259eb4ca8b719e560820a4cca969cc669e3c784c21d54e" Nov 02 22:00:03.036988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.036942 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:03.354984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.354871 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:03.507393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.483904 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" event={"ID":"ebed37fd-1beb-4f28-8381-45d6ecc6021b","Type":"ContainerStarted","Data":"ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6"} Nov 02 22:00:03.507393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.487838 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" event={"ID":"5751cfdb-44b0-4845-a8fb-e97695942dbc","Type":"ContainerStarted","Data":"7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd"} Nov 02 22:00:03.510803 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.510751 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerStarted","Data":"52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9"} Nov 02 22:00:03.799155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.748766 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" podStartSLOduration=3.748744549 podStartE2EDuration="3.748744549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.647690745 +0000 UTC m=+1542.609241065" watchObservedRunningTime="2026-11-02 22:00:03.748744549 +0000 UTC m=+1542.710294869" Nov 02 22:00:03.799155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.791019 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb"] Nov 02 22:00:03.871742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.871631 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:03.936628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.924958 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-caac72\"" Nov 02 22:00:03.936628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.925093 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-caac72-dockercfg-4vd6p\"" Nov 02 22:00:03.982607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:03.982559 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:04.102339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.102299 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmxr\" (UniqueName: \"kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr\") pod \"payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:04.208164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.207099 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb"] Nov 02 22:00:04.255177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.240374 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:04.255177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.240477 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmxr\" (UniqueName: \"kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr\") pod \"payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:04.255177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.242020 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:04.323181 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.317920 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" podStartSLOduration=4.3178995350000005 podStartE2EDuration="4.317899535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:03.912498893 +0000 UTC m=+1542.874049215" watchObservedRunningTime="2026-11-02 22:00:04.317899535 +0000 UTC m=+1543.279449855" Nov 02 22:00:04.345147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.342807 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmxr\" (UniqueName: \"kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr\") pod \"payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:04.524899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.524809 4285 generic.go:356] "Generic (PLEG): container finished" podID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerID="ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6" exitCode=60 Nov 02 22:00:04.525423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.524892 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" event={"ID":"ebed37fd-1beb-4f28-8381-45d6ecc6021b","Type":"ContainerDied","Data":"ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6"} Nov 02 22:00:04.525423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.524970 4285 scope.go:122] "RemoveContainer" containerID="21d90a8a5446377618feec2040bfd5433c19bf37da7a3f4d8182c00d4577bbf2" Nov 02 22:00:04.525423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.525401 4285 scope.go:122] "RemoveContainer" containerID="ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6" Nov 02 22:00:04.525904 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:04.525877 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-7rvc4_odh-ai-gateway-infra(ebed37fd-1beb-4f28-8381-45d6ecc6021b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" Nov 02 22:00:04.527521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.527496 4285 generic.go:356] "Generic (PLEG): container finished" podID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerID="7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd" exitCode=60 Nov 02 22:00:04.527640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.527579 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" event={"ID":"5751cfdb-44b0-4845-a8fb-e97695942dbc","Type":"ContainerDied","Data":"7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd"} Nov 02 22:00:04.529383 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.527923 4285 scope.go:122] "RemoveContainer" containerID="7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd" Nov 02 22:00:04.529383 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:04.528372 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv_odh-ai-gateway-infra(5751cfdb-44b0-4845-a8fb-e97695942dbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" Nov 02 22:00:04.530482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.530457 4285 generic.go:356] "Generic (PLEG): container finished" podID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerID="52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9" exitCode=60 Nov 02 22:00:04.530585 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.530543 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerDied","Data":"52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9"} Nov 02 22:00:04.531065 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.531046 4285 scope.go:122] "RemoveContainer" containerID="52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9" Nov 02 22:00:04.531503 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:04.531479 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx_odh-ai-gateway-infra(8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" Nov 02 22:00:04.540458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.540434 4285 scope.go:122] "RemoveContainer" containerID="3bc8cd8cb167c5f916a207f4e8aa2687ce548e106eb74c8e790ebaef3b7ba7f8" Nov 02 22:00:04.554885 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.554862 4285 scope.go:122] "RemoveContainer" containerID="b380bf8c271f64e227259eb4ca8b719e560820a4cca969cc669e3c784c21d54e" Nov 02 22:00:04.639508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:04.639479 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:05.023790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.023759 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb"] Nov 02 22:00:05.025972 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:05.025941 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01a427e_c877_44eb_bd0d_5d2012f93a8f.slice/crio-2ae1c919e719cd8b7f7055984d4bda037e49daf383f40a6e03bf84aeeba14e0b WatchSource:0}: Error finding container 2ae1c919e719cd8b7f7055984d4bda037e49daf383f40a6e03bf84aeeba14e0b: Status 404 returned error can't find the container with id 2ae1c919e719cd8b7f7055984d4bda037e49daf383f40a6e03bf84aeeba14e0b Nov 02 22:00:05.541724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.541690 4285 scope.go:122] "RemoveContainer" containerID="ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6" Nov 02 22:00:05.542224 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:05.542164 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-7rvc4_odh-ai-gateway-infra(ebed37fd-1beb-4f28-8381-45d6ecc6021b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" Nov 02 22:00:05.543366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.543335 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" event={"ID":"d01a427e-c877-44eb-bd0d-5d2012f93a8f","Type":"ContainerStarted","Data":"40fa8dca2769c65ad55239956ab1b05498a483fb9726d916065fb3b5c5ea24b5"} Nov 02 22:00:05.543522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.543390 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" event={"ID":"d01a427e-c877-44eb-bd0d-5d2012f93a8f","Type":"ContainerStarted","Data":"2ae1c919e719cd8b7f7055984d4bda037e49daf383f40a6e03bf84aeeba14e0b"} Nov 02 22:00:05.543522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.543418 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:05.545348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.545330 4285 scope.go:122] "RemoveContainer" containerID="7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd" Nov 02 22:00:05.545738 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:05.545712 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv_odh-ai-gateway-infra(5751cfdb-44b0-4845-a8fb-e97695942dbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" Nov 02 22:00:05.546992 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.546977 4285 scope.go:122] "RemoveContainer" containerID="52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9" Nov 02 22:00:05.547288 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:05.547268 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx_odh-ai-gateway-infra(8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" Nov 02 22:00:05.911951 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:05.911846 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" podStartSLOduration=2.911832001 podStartE2EDuration="2.911832001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:05.908318633 +0000 UTC m=+1544.869868987" watchObservedRunningTime="2026-11-02 22:00:05.911832001 +0000 UTC m=+1544.873382320" Nov 02 22:00:06.954711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:06.954680 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:07.239524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:07.239431 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:07.401263 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:07.401232 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:09.016871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.016836 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:09.553154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.547517 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2"] Nov 02 22:00:09.593148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.591572 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.639374 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.639332 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.740466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.687586 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2"] Nov 02 22:00:09.740466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.688161 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:09.887254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.887169 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d1c62e-openshift-default-dockercfg-7tmgm\"" Nov 02 22:00:09.932353 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.891106 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940526 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940623 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940694 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940742 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940788 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zt9\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940868 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.940997 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.941094 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:09.942153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:09.941809 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.038522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.038398 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.042965 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043099 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zt9\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043207 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043421 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043650 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043804 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043865 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.044543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.043926 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.047338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.047270 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.048470 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.048418 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.048714 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.048687 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.048870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.048829 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.049009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.048918 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.096922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.095638 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.214624 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.208460 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zt9\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.231183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.226389 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token\") pod \"e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.295986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.295948 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:10.729402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.729373 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2"] Nov 02 22:00:10.766929 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:10.766901 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f04c3b_044c_40a2_a010_cc06273a9103.slice/crio-776e49a310854581c2c5a33af56248b3e2192f497b20517c607c64f5b05c1ae5 WatchSource:0}: Error finding container 776e49a310854581c2c5a33af56248b3e2192f497b20517c607c64f5b05c1ae5: Status 404 returned error can't find the container with id 776e49a310854581c2c5a33af56248b3e2192f497b20517c607c64f5b05c1ae5 Nov 02 22:00:10.769813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.769766 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:10.769892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.769855 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:10.769892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.769886 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:10.904628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:10.904582 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:11.149091 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.148998 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc"] Nov 02 22:00:11.153870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.153847 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.215114 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.215082 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc"] Nov 02 22:00:11.294961 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.294926 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295005 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295072 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295228 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295263 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295302 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvgl\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295485 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295343 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295621 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.295809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.295782 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.397723 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.397690 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.397897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.397803 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.397913 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.397978 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398047 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398087 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398111 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398047 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398189 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvgl\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398258 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398281 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398391 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398613 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.398721 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.398701 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.400663 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.400600 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.400879 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.400859 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.454106 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.454064 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.454267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.454074 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvgl\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl\") pod \"maas-default-gateway-openshift-default-65c448b547-przjc\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.488356 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.488314 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:11.683709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.683633 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" event={"ID":"b7f04c3b-044c-40a2-a010-cc06273a9103","Type":"ContainerStarted","Data":"9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3"} Nov 02 22:00:11.683709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.683690 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" event={"ID":"b7f04c3b-044c-40a2-a010-cc06273a9103","Type":"ContainerStarted","Data":"776e49a310854581c2c5a33af56248b3e2192f497b20517c607c64f5b05c1ae5"} Nov 02 22:00:11.799934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.799882 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podStartSLOduration=2.799868113 podStartE2EDuration="2.799868113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:11.797392546 +0000 UTC m=+1550.758942878" watchObservedRunningTime="2026-11-02 22:00:11.799868113 +0000 UTC m=+1550.761418432" Nov 02 22:00:11.901541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.901503 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:11.987778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:11.987719 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc"] Nov 02 22:00:12.022788 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:12.018996 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce9f122_a121_4d82_bc6c_3ff4260d7187.slice/crio-8de4ef17725e6a81f27c0a73af45359c1803fda8bcddb44e9bacdd001da706c2 WatchSource:0}: Error finding container 8de4ef17725e6a81f27c0a73af45359c1803fda8bcddb44e9bacdd001da706c2: Status 404 returned error can't find the container with id 8de4ef17725e6a81f27c0a73af45359c1803fda8bcddb44e9bacdd001da706c2 Nov 02 22:00:12.022788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.022341 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:12.022788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.022424 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:12.022788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.022470 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:12.296203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.296097 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:12.301428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.301403 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:12.905033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.904996 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" event={"ID":"bce9f122-a121-4d82-bc6c-3ff4260d7187","Type":"ContainerStarted","Data":"546a06302adde7a7004039d0dfe04f9a1789487c57e1f30f953ab0131e48ffda"} Nov 02 22:00:12.905243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.905084 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:12.905243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.905110 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" event={"ID":"bce9f122-a121-4d82-bc6c-3ff4260d7187","Type":"ContainerStarted","Data":"8de4ef17725e6a81f27c0a73af45359c1803fda8bcddb44e9bacdd001da706c2"} Nov 02 22:00:12.906608 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:12.906586 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:00:13.026841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:13.026790 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podStartSLOduration=2.026775646 podStartE2EDuration="2.026775646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:13.020330402 +0000 UTC m=+1551.981880750" watchObservedRunningTime="2026-11-02 22:00:13.026775646 +0000 UTC m=+1551.988325965" Nov 02 22:00:13.507145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:13.496354 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:13.507145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:13.498300 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:13.507145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:13.498358 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:14.182152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:14.166899 4285 scope.go:122] "RemoveContainer" containerID="ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6" Nov 02 22:00:14.182152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:14.168028 4285 scope.go:122] "RemoveContainer" containerID="7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd" Nov 02 22:00:14.495975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:14.490228 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:14.495975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:14.490286 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:14.601315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:14.601102 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:14.925621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:14.925590 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:15.101426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:15.101388 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:15.173148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:15.170510 4285 scope.go:122] "RemoveContainer" containerID="52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9" Nov 02 22:00:15.493151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:15.490618 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:15.493151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:15.490675 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:16.052103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.052016 4285 generic.go:356] "Generic (PLEG): container finished" podID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerID="cccf8a803e9fe907b08ab2a564d1d24203a6896ef37d53324360722e6fa8c250" exitCode=60 Nov 02 22:00:16.057762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.057731 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" event={"ID":"5751cfdb-44b0-4845-a8fb-e97695942dbc","Type":"ContainerDied","Data":"cccf8a803e9fe907b08ab2a564d1d24203a6896ef37d53324360722e6fa8c250"} Nov 02 22:00:16.057932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.057914 4285 scope.go:122] "RemoveContainer" containerID="7e1b45f729bae1d3051075f18ff79832fdfb7650b830a25a541f134a5af037fd" Nov 02 22:00:16.058629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.058608 4285 scope.go:122] "RemoveContainer" containerID="cccf8a803e9fe907b08ab2a564d1d24203a6896ef37d53324360722e6fa8c250" Nov 02 22:00:16.059250 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:16.059222 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv_odh-ai-gateway-infra(5751cfdb-44b0-4845-a8fb-e97695942dbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" Nov 02 22:00:16.082157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.080695 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerStarted","Data":"d817e8b6d340fc37babd450402464b37649785e67f7e7e3fc101384f77a8ebaa"} Nov 02 22:00:16.083892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.082812 4285 generic.go:356] "Generic (PLEG): container finished" podID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerID="f9b575ccd0643a1eba1a23b0df8e46e1483db755bc919d6b234e8bac16d7512e" exitCode=60 Nov 02 22:00:16.083892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.082927 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" event={"ID":"ebed37fd-1beb-4f28-8381-45d6ecc6021b","Type":"ContainerDied","Data":"f9b575ccd0643a1eba1a23b0df8e46e1483db755bc919d6b234e8bac16d7512e"} Nov 02 22:00:16.083892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.083406 4285 scope.go:122] "RemoveContainer" containerID="f9b575ccd0643a1eba1a23b0df8e46e1483db755bc919d6b234e8bac16d7512e" Nov 02 22:00:16.083892 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:16.083839 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894280-7rvc4_odh-ai-gateway-infra(ebed37fd-1beb-4f28-8381-45d6ecc6021b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" Nov 02 22:00:16.103316 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.103213 4285 scope.go:122] "RemoveContainer" containerID="ac0cc108e63e6a4b10b3b5a111b623a8c6b7f58a84b65c8a196817859e0dd6c6" Nov 02 22:00:16.127941 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:16.124327 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:00:16.500149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.497524 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:16.500149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.497588 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:16.636156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:16.618351 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:00:17.474181 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.445107 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv"] Nov 02 22:00:17.496156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.493581 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:17.496156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.493667 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:17.582293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.581375 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4"] Nov 02 22:00:17.601204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.597814 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 22:00:17.601204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.597962 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 22:00:17.607154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.603555 4285 generic.go:356] "Generic (PLEG): container finished" podID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerID="d817e8b6d340fc37babd450402464b37649785e67f7e7e3fc101384f77a8ebaa" exitCode=60 Nov 02 22:00:17.607154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.603784 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerDied","Data":"d817e8b6d340fc37babd450402464b37649785e67f7e7e3fc101384f77a8ebaa"} Nov 02 22:00:17.607154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.603847 4285 scope.go:122] "RemoveContainer" containerID="52e65e71ecfb24f311d9c54f4fa8d43348c781b830d778bdeeea990417952dc9" Nov 02 22:00:17.607154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.604369 4285 scope.go:122] "RemoveContainer" containerID="d817e8b6d340fc37babd450402464b37649785e67f7e7e3fc101384f77a8ebaa" Nov 02 22:00:17.607154 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:17.604825 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx_odh-ai-gateway-infra(8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" Nov 02 22:00:17.724150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.722982 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 22:00:17.746566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.729472 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:17.838150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.795283 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 22:00:17.895867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:17.895807 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx"] Nov 02 22:00:18.505151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.504047 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:18.505151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.504114 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:18.734147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.733312 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:18.870199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.869282 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca\") pod \"5751cfdb-44b0-4845-a8fb-e97695942dbc\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " Nov 02 22:00:18.870199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.869529 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l\" (UniqueName: \"kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l\") pod \"5751cfdb-44b0-4845-a8fb-e97695942dbc\" (UID: \"5751cfdb-44b0-4845-a8fb-e97695942dbc\") " Nov 02 22:00:18.883149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.876056 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca" pod "5751cfdb-44b0-4845-a8fb-e97695942dbc" (UID: "5751cfdb-44b0-4845-a8fb-e97695942dbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:18.984150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:18.977847 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751cfdb-44b0-4845-a8fb-e97695942dbc-maas-api-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.029303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.001738 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l" pod "5751cfdb-44b0-4845-a8fb-e97695942dbc" (UID: "5751cfdb-44b0-4845-a8fb-e97695942dbc"). InnerVolumeSpecName "kube-api-access-twr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:19.139231 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.100335 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twr5l\" (UniqueName: \"kubernetes.io/projected/5751cfdb-44b0-4845-a8fb-e97695942dbc-kube-api-access-twr5l\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.168550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.164960 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" event={"ID":"5751cfdb-44b0-4845-a8fb-e97695942dbc","Type":"ContainerDied","Data":"78a1cd9ccde08620d7a05afaa7677f4656a7140d8c0ee1c13a7bb57e645135c4"} Nov 02 22:00:19.168550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.165038 4285 scope.go:122] "RemoveContainer" containerID="cccf8a803e9fe907b08ab2a564d1d24203a6896ef37d53324360722e6fa8c250" Nov 02 22:00:19.168550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.165164 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv" Nov 02 22:00:19.486163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.466935 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:19.491960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.490253 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:19.491960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.490325 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:19.515108 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.512972 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca\") pod \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " Nov 02 22:00:19.515108 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.513027 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c\" (UniqueName: \"kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c\") pod \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\" (UID: \"ebed37fd-1beb-4f28-8381-45d6ecc6021b\") " Nov 02 22:00:19.532157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.515825 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca" pod "ebed37fd-1beb-4f28-8381-45d6ecc6021b" (UID: "ebed37fd-1beb-4f28-8381-45d6ecc6021b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:19.689401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.689294 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:19.690265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.690231 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:19.691262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.690720 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 22:00:19.720240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.697933 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebed37fd-1beb-4f28-8381-45d6ecc6021b-maas-api-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.720240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.714453 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c" pod "ebed37fd-1beb-4f28-8381-45d6ecc6021b" (UID: "ebed37fd-1beb-4f28-8381-45d6ecc6021b"). InnerVolumeSpecName "kube-api-access-58f9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:19.720572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.720539 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:19.774147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.750756 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:19.783192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.754317 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 22:00:19.788738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.788701 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:19.800043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.799759 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f9c\" (UniqueName: \"kubernetes.io/projected/ebed37fd-1beb-4f28-8381-45d6ecc6021b-kube-api-access-58f9c\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.811571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.811535 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:19.892215 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.892183 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv"] Nov 02 22:00:19.900385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:19.900348 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9cc56b-29894280-tr9nv"] Nov 02 22:00:20.506854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.503955 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:20.506854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.504013 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:20.601043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.601008 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" Nov 02 22:00:20.601552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.601433 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4" event={"ID":"ebed37fd-1beb-4f28-8381-45d6ecc6021b","Type":"ContainerDied","Data":"6e7893533c146dc64b9d2879134ac0efd66e144c79aa914196047fe05da5d396"} Nov 02 22:00:20.601552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.601523 4285 scope.go:122] "RemoveContainer" containerID="f9b575ccd0643a1eba1a23b0df8e46e1483db755bc919d6b234e8bac16d7512e" Nov 02 22:00:20.604834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.604747 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:20.605235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.605183 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca\") pod \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " Nov 02 22:00:20.606073 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.606026 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca" pod "8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" (UID: "8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:20.706789 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.706749 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4\" (UniqueName: \"kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4\") pod \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\" (UID: \"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0\") " Nov 02 22:00:20.707946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.707913 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-maas-api-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.708912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.708887 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4" pod "8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" (UID: "8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0"). InnerVolumeSpecName "kube-api-access-z8zm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:20.824661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.824631 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zm4\" (UniqueName: \"kubernetes.io/projected/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0-kube-api-access-z8zm4\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.864528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.864442 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:20.884380 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.884349 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4"] Nov 02 22:00:20.888525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:20.888500 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894280-7rvc4"] Nov 02 22:00:21.294148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.291206 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:21.578250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.525278 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:21.578250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.525339 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:21.653234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.652617 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" Nov 02 22:00:21.897876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.897790 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" path="/var/lib/kubelet/pods/5751cfdb-44b0-4845-a8fb-e97695942dbc/volumes" Nov 02 22:00:21.898924 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.898898 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" path="/var/lib/kubelet/pods/ebed37fd-1beb-4f28-8381-45d6ecc6021b/volumes" Nov 02 22:00:21.899715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.899696 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4"] Nov 02 22:00:21.900906 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.900885 4285 cpu_manager.go:414] "containerMap: removing container" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:21.901016 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901003 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:21.901108 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901098 4285 cpu_manager.go:414] "containerMap: removing container" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.901388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901373 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.901499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901486 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.901602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901589 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.901690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901680 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.901768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901756 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.901863 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901851 4285 cpu_manager.go:414] "containerMap: removing container" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.901944 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.901934 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.902023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902014 4285 cpu_manager.go:414] "containerMap: removing container" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:21.902094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902084 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:21.902216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902205 4285 cpu_manager.go:414] "containerMap: removing container" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.902320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902309 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.902570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902558 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.902655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902646 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.902731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902723 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.902805 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902796 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:21.902875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902867 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:21.902950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.902942 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:21.903030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.903021 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:21.918498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.918469 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx" event={"ID":"8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0","Type":"ContainerDied","Data":"2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7"} Nov 02 22:00:21.918608 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.918522 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4"] Nov 02 22:00:21.918608 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.918580 4285 scope.go:122] "RemoveContainer" containerID="d817e8b6d340fc37babd450402464b37649785e67f7e7e3fc101384f77a8ebaa" Nov 02 22:00:21.918845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.918827 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:21.973646 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.972458 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d1c62e\"" Nov 02 22:00:21.974091 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.974040 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d1c62e-dockercfg-mz9l5\"" Nov 02 22:00:21.984736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.984678 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx"] Nov 02 22:00:21.993241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.993197 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ffe386-29894280-cxjgx"] Nov 02 22:00:21.999638 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.999605 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:21.999783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:21.999725 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzz29\" (UniqueName: \"kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29\") pod \"payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:22.108156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.105637 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzz29\" (UniqueName: \"kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29\") pod \"payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:22.108156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.106073 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:22.108156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.107499 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:22.247873 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.246782 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzz29\" (UniqueName: \"kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29\") pod \"payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:22.490035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.489996 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 22:00:22.490214 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.490061 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 22:00:22.542187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.541943 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:22.858738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:22.858658 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:23.025050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:23.025014 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4"] Nov 02 22:00:23.180115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:23.180075 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" path="/var/lib/kubelet/pods/8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0/volumes" Nov 02 22:00:23.733983 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:23.733543 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" event={"ID":"b2c3a9a9-5ffd-4638-9986-36ff5876e38f","Type":"ContainerStarted","Data":"8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35"} Nov 02 22:00:23.733983 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:23.733609 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" event={"ID":"b2c3a9a9-5ffd-4638-9986-36ff5876e38f","Type":"ContainerStarted","Data":"924727cab285b1a6e6689b165398f7f0fb762fb3aefd3136128e588f982cf7e6"} Nov 02 22:00:23.735260 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:23.735177 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:23.855902 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:23.855844 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" podStartSLOduration=2.855824596 podStartE2EDuration="2.855824596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:23.845669377 +0000 UTC m=+1562.807219697" watchObservedRunningTime="2026-11-02 22:00:23.855824596 +0000 UTC m=+1562.817374917" Nov 02 22:00:24.203783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:24.202774 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:24.204301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:24.203923 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:24.204902 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:24.204873 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:00:24.365297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:24.365260 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:00:24.365536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:24.365325 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:00:26.767947 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:26.767915 4285 generic.go:356] "Generic (PLEG): container finished" podID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerID="02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb" exitCode=0 Nov 02 22:00:26.769307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:26.767982 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" event={"ID":"ea7abda0-213a-41bb-be21-4f6916d45cfe","Type":"ContainerDied","Data":"02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb"} Nov 02 22:00:26.769307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:26.768537 4285 scope.go:122] "RemoveContainer" containerID="02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb" Nov 02 22:00:26.770362 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:26.770335 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:26.770437 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:26.770401 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:26.770437 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:26.770431 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:27.777298 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.777266 4285 generic.go:356] "Generic (PLEG): container finished" podID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerID="30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80" exitCode=0 Nov 02 22:00:27.777714 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.777333 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" event={"ID":"db124196-7f89-46f5-a6cc-21782b23a2bc","Type":"ContainerDied","Data":"30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80"} Nov 02 22:00:27.777865 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.777849 4285 scope.go:122] "RemoveContainer" containerID="30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80" Nov 02 22:00:27.779548 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.779526 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" event={"ID":"ea7abda0-213a-41bb-be21-4f6916d45cfe","Type":"ContainerStarted","Data":"97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2"} Nov 02 22:00:27.779820 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.779793 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:27.779907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.779877 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:27.779969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:27.779926 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:28.074656 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:28.074568 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 22:00:28.075803 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:28.075775 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:00:28.075942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:28.075829 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 22:00:28.208148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:28.198903 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:28.798736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:28.798695 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" event={"ID":"db124196-7f89-46f5-a6cc-21782b23a2bc","Type":"ContainerStarted","Data":"ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7"} Nov 02 22:00:29.073788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:29.073709 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:00:29.073788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:29.073758 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 22:00:29.162541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:29.162501 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 22:00:29.162899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:29.162866 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:00:29.163071 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:29.163046 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:00:30.074393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:30.074354 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:00:30.074785 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:30.074407 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 22:00:30.160584 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:30.160543 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:00:30.171265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:30.171222 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:00:30.766266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:30.766233 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 22:00:31.074091 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:31.073988 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:00:31.074091 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:31.074042 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 22:00:31.158525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:31.158485 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:00:31.159107 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:31.159076 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:00:31.975276 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:31.975219 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:32.074666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:32.074636 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:00:32.074890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:32.074869 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 22:00:32.161157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:32.159496 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:00:32.161157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:32.159560 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:00:32.359731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:32.359617 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:33.074272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:33.074235 4285 patch_prober.go:29] interesting pod/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:00:33.074538 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:33.074507 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 22:00:33.259117 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:33.259078 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:33.260025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:33.259998 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:00:33.260243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:33.260216 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:00:33.426149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:33.422280 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:34.039672 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:34.039627 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:34.100621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:34.099778 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 22:00:34.102052 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:34.102027 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 22:00:34.158201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:34.158168 4285 patch_prober.go:29] interesting pod/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:00:34.158367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:34.158229 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 22:00:35.224691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:35.224657 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 22:00:35.228540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:35.228514 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 22:00:35.229878 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:35.229857 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 22:00:35.761178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:35.761114 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:00:36.278663 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:36.278600 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:36.435623 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:36.435582 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:37.767841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:37.767804 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:38.008149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:37.998097 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:38.260244 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:38.260165 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:38.544152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:38.544032 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:41.246008 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:41.245965 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:41.246526 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:41.246263 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:41.991804 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:41.991764 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:43.109484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:43.109440 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:46.007737 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:46.007700 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:00:46.094392 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:46.094354 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm"] Nov 02 22:00:46.208523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:46.208486 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj"] Nov 02 22:00:46.570415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:46.553839 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:46.711112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:46.710967 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:47.835148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:47.832254 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:49.099746 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:49.099688 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:51.095600 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.095555 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" podUID="ee045427-4e77-46e7-b0e0-9a5cd4adb600" containerName="payload-processing" containerID="cri-o://fe95b813c2f1d49214de39500cf25699cf36f853fcab485cfa773396b94eb8a8" gracePeriod=25 Nov 02 22:00:51.209427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.209389 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" podUID="0a0ab3a3-1b20-437a-8225-c58d09bc6a14" containerName="payload-pre-processing" containerID="cri-o://5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a" gracePeriod=25 Nov 02 22:00:51.324249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.324158 4285 generic.go:356] "Generic (PLEG): container finished" podID="ee045427-4e77-46e7-b0e0-9a5cd4adb600" containerID="fe95b813c2f1d49214de39500cf25699cf36f853fcab485cfa773396b94eb8a8" exitCode=0 Nov 02 22:00:51.324334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.324305 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" event={"ID":"ee045427-4e77-46e7-b0e0-9a5cd4adb600","Type":"ContainerDied","Data":"fe95b813c2f1d49214de39500cf25699cf36f853fcab485cfa773396b94eb8a8"} Nov 02 22:00:51.416548 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.416521 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 22:00:51.515838 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.515808 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 22:00:51.608725 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.608686 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp\" (UniqueName: \"kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp\") pod \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " Nov 02 22:00:51.608894 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.608758 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume\") pod \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\" (UID: \"ee045427-4e77-46e7-b0e0-9a5cd4adb600\") " Nov 02 22:00:51.609251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.609221 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume" pod "ee045427-4e77-46e7-b0e0-9a5cd4adb600" (UID: "ee045427-4e77-46e7-b0e0-9a5cd4adb600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:51.609552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.609532 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045427-4e77-46e7-b0e0-9a5cd4adb600-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:51.610919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.610897 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp" pod "ee045427-4e77-46e7-b0e0-9a5cd4adb600" (UID: "ee045427-4e77-46e7-b0e0-9a5cd4adb600"). InnerVolumeSpecName "kube-api-access-mmxwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:51.710367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.710331 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr\" (UniqueName: \"kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr\") pod \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " Nov 02 22:00:51.710593 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.710561 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume\") pod \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\" (UID: \"0a0ab3a3-1b20-437a-8225-c58d09bc6a14\") " Nov 02 22:00:51.711016 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.710988 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume" pod "0a0ab3a3-1b20-437a-8225-c58d09bc6a14" (UID: "0a0ab3a3-1b20-437a-8225-c58d09bc6a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:51.711595 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.711569 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxwp\" (UniqueName: \"kubernetes.io/projected/ee045427-4e77-46e7-b0e0-9a5cd4adb600-kube-api-access-mmxwp\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:51.711701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.711601 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:51.713006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.712978 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr" pod "0a0ab3a3-1b20-437a-8225-c58d09bc6a14" (UID: "0a0ab3a3-1b20-437a-8225-c58d09bc6a14"). InnerVolumeSpecName "kube-api-access-6z6kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:51.813971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:51.813927 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6kr\" (UniqueName: \"kubernetes.io/projected/0a0ab3a3-1b20-437a-8225-c58d09bc6a14-kube-api-access-6z6kr\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:00:52.006479 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:52.006398 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:00:52.344872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.344777 4285 generic.go:356] "Generic (PLEG): container finished" podID="0a0ab3a3-1b20-437a-8225-c58d09bc6a14" containerID="5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a" exitCode=0 Nov 02 22:00:52.344872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.344837 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" Nov 02 22:00:52.344872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.344857 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" event={"ID":"0a0ab3a3-1b20-437a-8225-c58d09bc6a14","Type":"ContainerDied","Data":"5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a"} Nov 02 22:00:52.345597 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.344937 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj" event={"ID":"0a0ab3a3-1b20-437a-8225-c58d09bc6a14","Type":"ContainerDied","Data":"8e82424a927aeae12a04e30b36c71890a446bfa8095f8fade0e8e40a52cc95ab"} Nov 02 22:00:52.345597 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.344960 4285 scope.go:122] "RemoveContainer" containerID="5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a" Nov 02 22:00:52.346952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.346931 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" event={"ID":"ee045427-4e77-46e7-b0e0-9a5cd4adb600","Type":"ContainerDied","Data":"be690efb63098806eea7a2ed6d338b9f45078287682f703523c6d840d1c871d8"} Nov 02 22:00:52.347059 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.346960 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm" Nov 02 22:00:52.357661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.357643 4285 scope.go:122] "RemoveContainer" containerID="5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a" Nov 02 22:00:52.357973 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:52.357950 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a\": container with ID starting with 5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a not found: ID does not exist" containerID="5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a" Nov 02 22:00:52.358080 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.357980 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a"} err="failed to get container status \"5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a\": rpc error: code = NotFound desc = could not find container \"5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a\": container with ID starting with 5c8fdcf0544e790c3f7077252968eb173626b9b281599e95802e249dc4438a4a not found: ID does not exist" Nov 02 22:00:52.358080 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.357998 4285 scope.go:122] "RemoveContainer" containerID="fe95b813c2f1d49214de39500cf25699cf36f853fcab485cfa773396b94eb8a8" Nov 02 22:00:52.509545 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.509511 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj"] Nov 02 22:00:52.558870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.558840 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4e90e53-dfd8998cf-nfgfj"] Nov 02 22:00:52.667168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.667046 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm"] Nov 02 22:00:52.671247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.671222 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4e90e53-686d8f55f-m7dsm"] Nov 02 22:00:52.704282 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:52.704256 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:53.174968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:53.174937 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0ab3a3-1b20-437a-8225-c58d09bc6a14" path="/var/lib/kubelet/pods/0a0ab3a3-1b20-437a-8225-c58d09bc6a14/volumes" Nov 02 22:00:53.175423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:53.175407 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee045427-4e77-46e7-b0e0-9a5cd4adb600" path="/var/lib/kubelet/pods/ee045427-4e77-46e7-b0e0-9a5cd4adb600/volumes" Nov 02 22:00:57.126288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.126254 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv"] Nov 02 22:00:57.127225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127202 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:57.127225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127224 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebed37fd-1beb-4f28-8381-45d6ecc6021b" containerName="cleanup" Nov 02 22:00:57.127414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127257 4285 cpu_manager.go:414] "containerMap: removing container" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:57.127414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127265 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:57.127414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127294 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ee045427-4e77-46e7-b0e0-9a5cd4adb600" containerName="payload-processing" Nov 02 22:00:57.127414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127302 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee045427-4e77-46e7-b0e0-9a5cd4adb600" containerName="payload-processing" Nov 02 22:00:57.127414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127329 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0a0ab3a3-1b20-437a-8225-c58d09bc6a14" containerName="payload-pre-processing" Nov 02 22:00:57.127414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127336 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0ab3a3-1b20-437a-8225-c58d09bc6a14" containerName="payload-pre-processing" Nov 02 22:00:57.127701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127513 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0ab3a3-1b20-437a-8225-c58d09bc6a14" containerName="payload-pre-processing" Nov 02 22:00:57.127701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127531 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="8751ec1f-685c-4b94-b4a1-6c3fedcd2aa0" containerName="cleanup" Nov 02 22:00:57.127701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127554 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="5751cfdb-44b0-4845-a8fb-e97695942dbc" containerName="cleanup" Nov 02 22:00:57.127701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.127563 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee045427-4e77-46e7-b0e0-9a5cd4adb600" containerName="payload-processing" Nov 02 22:00:57.131269 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.131249 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.190706 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.190400 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-caac72-serving-cert\"" Nov 02 22:00:57.194042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.194008 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdpv\" (UniqueName: \"kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.194193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.194168 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.194270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.194207 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.194764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.194342 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.247525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.247496 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv"] Nov 02 22:00:57.295564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.295530 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.295787 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.295768 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdpv\" (UniqueName: \"kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.296118 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.296095 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.296365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.296347 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.296653 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:57.296551 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-serving-cert: secret "maas-api-e2e-shared-a-w5-caac72-serving-cert" not found Nov 02 22:00:57.296864 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:00:57.296850 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls podName:8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9 nodeName:}" failed. No retries permitted until 2026-11-02 22:00:57.796828495 +0000 UTC m=+1596.758378798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls") pod "maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" (UID: "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9") : secret "maas-api-e2e-shared-a-w5-caac72-serving-cert" not found Nov 02 22:00:57.297251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.297230 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.299845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.299820 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.352927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.352894 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdpv\" (UniqueName: \"kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.357755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.357703 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:57.514401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.513967 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:57.667781 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.667744 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:57.852604 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.852523 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:57.854958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:57.854931 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:58.044288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:58.044259 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:58.047705 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:58.047681 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct"] Nov 02 22:00:58.052002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:58.051978 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4580c2d6-4bsct"] Nov 02 22:00:58.469603 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:00:58.469570 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1e9c97_e22c_41d3_890e_71b4d7b2d6e9.slice/crio-edcee5ff226267a68d62e84b845e23b0db8558346411a4d3f8ab19900936ff77 WatchSource:0}: Error finding container edcee5ff226267a68d62e84b845e23b0db8558346411a4d3f8ab19900936ff77: Status 404 returned error can't find the container with id edcee5ff226267a68d62e84b845e23b0db8558346411a4d3f8ab19900936ff77 Nov 02 22:00:58.470105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:58.470083 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv"] Nov 02 22:00:59.176381 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:59.176347 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388fc395-c211-4f65-a6ac-9c23be0c6b91" path="/var/lib/kubelet/pods/388fc395-c211-4f65-a6ac-9c23be0c6b91/volumes" Nov 02 22:00:59.482410 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:59.482371 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" event={"ID":"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9","Type":"ContainerStarted","Data":"c381ccd61ca628b7e8859500f5a37275937d41b57f5633dc5d31abf7e02ff955"} Nov 02 22:00:59.482815 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:59.482445 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" event={"ID":"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9","Type":"ContainerStarted","Data":"edcee5ff226267a68d62e84b845e23b0db8558346411a4d3f8ab19900936ff77"} Nov 02 22:00:59.482815 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:59.482716 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:00:59.635727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:00:59.635677 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" podStartSLOduration=1.722617782 podStartE2EDuration="2.635661399s" totalImagesPullingTime="913.043617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:59.631012762 +0000 UTC m=+1598.592563082" watchObservedRunningTime="2026-11-02 22:00:59.635661399 +0000 UTC m=+1598.597211718" Nov 02 22:01:02.102047 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:02.102007 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache]" Nov 02 22:01:02.254975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:02.254936 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:03.989260 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:03.989220 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache]" Nov 02 22:01:05.506673 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:05.506636 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:01:06.980139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:06.980085 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:09.890413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:09.890376 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:10.048489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:10.048438 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:10.237859 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:10.237791 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:11.001436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.001402 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99"] Nov 02 22:01:11.001947 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.001624 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" podUID="2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" containerName="maas-api" containerID="cri-o://633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32" gracePeriod=30 Nov 02 22:01:11.147019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.146943 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:11.309041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.308810 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:11.488145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.482416 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 22:01:11.492165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.490357 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:11.875256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.808469 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz\" (UniqueName: \"kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz\") pod \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " Nov 02 22:01:11.875256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.808683 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs\") pod \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " Nov 02 22:01:11.875256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.808760 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca\") pod \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " Nov 02 22:01:11.875256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.808815 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls\") pod \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\" (UID: \"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e\") " Nov 02 22:01:11.905151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.875831 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca" pod "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" (UID: "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:11.930119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.916161 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:11.930119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.917453 4285 generic.go:356] "Generic (PLEG): container finished" podID="2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" containerID="633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32" exitCode=0 Nov 02 22:01:11.930119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.917605 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" event={"ID":"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e","Type":"ContainerDied","Data":"633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32"} Nov 02 22:01:11.930119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.917661 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" event={"ID":"2fb54226-a6c6-4cb6-bd5f-98dbba37d05e","Type":"ContainerDied","Data":"15e8fabcbb30f473c14aef886137b12a0143df8516dccff3de55a9435f64b273"} Nov 02 22:01:11.930119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.917691 4285 scope.go:122] "RemoveContainer" containerID="633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32" Nov 02 22:01:11.930119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.917894 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99" Nov 02 22:01:11.960142 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:11.960088 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls" pod "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" (UID: "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:12.055094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.053562 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:12.081163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.056014 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs" pod "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" (UID: "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:01:12.081163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.056476 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz" pod "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" (UID: "2fb54226-a6c6-4cb6-bd5f-98dbba37d05e"). InnerVolumeSpecName "kube-api-access-rwqzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:12.164192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.160560 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqzz\" (UniqueName: \"kubernetes.io/projected/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-kube-api-access-rwqzz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:12.164192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.160599 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:12.208623 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.208408 4285 scope.go:122] "RemoveContainer" containerID="633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32" Nov 02 22:01:12.213608 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:12.213575 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32\": container with ID starting with 633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32 not found: ID does not exist" containerID="633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32" Nov 02 22:01:12.213847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.213819 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32"} err="failed to get container status \"633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32\": rpc error: code = NotFound desc = could not find container \"633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32\": container with ID starting with 633e45dc0d13dd413c0fea370f6574e67ac0993c907b90b43f53d8d27650ed32 not found: ID does not exist" Nov 02 22:01:12.703810 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.702269 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99"] Nov 02 22:01:12.707805 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.707776 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34089f5b-7bdbc74dc4-dpd99"] Nov 02 22:01:12.758492 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:12.758454 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb54226_a6c6_4cb6_bd5f_98dbba37d05e.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:01:12.827710 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.827677 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:12.838401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.838369 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn"] Nov 02 22:01:12.839882 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.839855 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" containerName="maas-api" Nov 02 22:01:12.840032 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.840018 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" containerName="maas-api" Nov 02 22:01:12.840353 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.840336 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" containerName="maas-api" Nov 02 22:01:12.847947 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.847930 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.896997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.896969 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.897276 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.897256 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklbm\" (UniqueName: \"kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.897340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.897307 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.897433 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.897411 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.899712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.899690 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d1c62e-serving-cert\"" Nov 02 22:01:12.903852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.903831 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn"] Nov 02 22:01:12.998719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.998648 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.998856 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.998804 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklbm\" (UniqueName: \"kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.998856 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:12.998816 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-serving-cert: secret "maas-api-e2e-shared-a-w3-d1c62e-serving-cert" not found Nov 02 22:01:12.998856 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.998843 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.998955 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:12.998899 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls podName:65d8a917-ecae-4593-b818-7a296d45f8fd nodeName:}" failed. No retries permitted until 2026-11-02 22:01:13.49887881 +0000 UTC m=+1612.460429131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls") pod "maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" (UID: "65d8a917-ecae-4593-b818-7a296d45f8fd") : secret "maas-api-e2e-shared-a-w3-d1c62e-serving-cert" not found Nov 02 22:01:12.998955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.998939 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:12.999870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:12.999842 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:13.001477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.001456 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:13.050650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.050620 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:13.055946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.055921 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklbm\" (UniqueName: \"kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:13.226652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.226574 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:13.284169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.266204 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb54226-a6c6-4cb6-bd5f-98dbba37d05e" path="/var/lib/kubelet/pods/2fb54226-a6c6-4cb6-bd5f-98dbba37d05e/volumes" Nov 02 22:01:13.431174 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.425574 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:13.566202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.562192 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:13.604145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.602894 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:13.762905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.762881 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:13.860135 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.860031 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk"] Nov 02 22:01:13.860576 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.860542 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" containerID="cri-o://ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7" gracePeriod=30 Nov 02 22:01:13.885958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:13.884883 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:14.509183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:14.506755 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn"] Nov 02 22:01:14.681495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:14.681453 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:15.523150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:15.515734 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" event={"ID":"65d8a917-ecae-4593-b818-7a296d45f8fd","Type":"ContainerStarted","Data":"b939e0efb601f3abdb504ce4445b7955802f7e9806fb8934ff85ded440082a41"} Nov 02 22:01:15.887744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:15.887706 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:15.989207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:15.983707 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh"] Nov 02 22:01:15.989207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:15.984040 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" containerID="cri-o://ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b" gracePeriod=30 Nov 02 22:01:16.076209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.076168 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:16.170201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.139720 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7"] Nov 02 22:01:16.170201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.146329 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" containerID="cri-o://7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc" gracePeriod=30 Nov 02 22:01:16.170201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.155958 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4"] Nov 02 22:01:16.170201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.156044 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx"] Nov 02 22:01:16.170201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.156321 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" containerID="cri-o://efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd" gracePeriod=30 Nov 02 22:01:16.170201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.156588 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" containerID="cri-o://ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224" gracePeriod=30 Nov 02 22:01:16.204450 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:16.204417 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:01:16.651498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.650211 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" event={"ID":"65d8a917-ecae-4593-b818-7a296d45f8fd","Type":"ContainerStarted","Data":"4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e"} Nov 02 22:01:16.651498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.651294 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:16.766455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:16.766407 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" podStartSLOduration=3.671603039 podStartE2EDuration="4.766392331s" totalImagesPullingTime="1.094789292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:16.762698292 +0000 UTC m=+1615.724248651" watchObservedRunningTime="2026-11-02 22:01:16.766392331 +0000 UTC m=+1615.727942651" Nov 02 22:01:18.631443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:18.631406 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:19.046963 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:19.046926 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8751ec1f_685c_4b94_b4a1_6c3fedcd2aa0.slice/crio-2e05cbaea35903a8d65d0f5b51dda173fbd5584e11c7ba6c7ebad0128be8b7c7\": RecentStats: unable to find data in memory cache]" Nov 02 22:01:19.160930 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.160861 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 22:01:19.252587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252558 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.252754 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252660 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.252754 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252702 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.252855 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252767 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.252855 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252809 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.252954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252912 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.253014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252954 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.253014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.252999 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.253138 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.253111 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy\") pod \"db124196-7f89-46f5-a6cc-21782b23a2bc\" (UID: \"db124196-7f89-46f5-a6cc-21782b23a2bc\") " Nov 02 22:01:19.253206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.253145 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:19.253206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.253151 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.253206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.253193 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.253380 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.253312 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.253522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.253497 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.254045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.254019 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.254167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.254049 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.254167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.254065 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.254167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.254082 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db124196-7f89-46f5-a6cc-21782b23a2bc-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.254167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.254096 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.255423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.255396 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "kube-api-access-5xmwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:19.255505 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.255427 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:19.255602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.255578 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:19.255731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.255717 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy" pod "db124196-7f89-46f5-a6cc-21782b23a2bc" (UID: "db124196-7f89-46f5-a6cc-21782b23a2bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:19.355570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.355528 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.355570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.355570 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmwr\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-kube-api-access-5xmwr\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.355839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.355588 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.355839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.355604 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db124196-7f89-46f5-a6cc-21782b23a2bc-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:19.637296 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.637257 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:01:19.637690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.637309 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:19.747183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.747107 4285 generic.go:356] "Generic (PLEG): container finished" podID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerID="ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7" exitCode=0 Nov 02 22:01:19.747183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.747177 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" event={"ID":"db124196-7f89-46f5-a6cc-21782b23a2bc","Type":"ContainerDied","Data":"ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7"} Nov 02 22:01:19.747416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.747189 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" Nov 02 22:01:19.747416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.747240 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk" event={"ID":"db124196-7f89-46f5-a6cc-21782b23a2bc","Type":"ContainerDied","Data":"343e1c5b6c00f34d98881dac4a50636230fe05d189e008482c59b01ec9646ef2"} Nov 02 22:01:19.747416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.747269 4285 scope.go:122] "RemoveContainer" containerID="ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7" Nov 02 22:01:19.760934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.760881 4285 scope.go:122] "RemoveContainer" containerID="30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80" Nov 02 22:01:19.771187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.771164 4285 scope.go:122] "RemoveContainer" containerID="ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7" Nov 02 22:01:19.771505 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:19.771485 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7\": container with ID starting with ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7 not found: ID does not exist" containerID="ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7" Nov 02 22:01:19.771595 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.771509 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7"} err="failed to get container status \"ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7\": rpc error: code = NotFound desc = could not find container \"ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7\": container with ID starting with ba4e2204e1b2b8cd57c1ab0293caf4dffc8be51cb3b89509d73e8a983ffb2ae7 not found: ID does not exist" Nov 02 22:01:19.771595 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.771531 4285 scope.go:122] "RemoveContainer" containerID="30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80" Nov 02 22:01:19.771800 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:19.771775 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80\": container with ID starting with 30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80 not found: ID does not exist" containerID="30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80" Nov 02 22:01:19.771866 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.771807 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80"} err="failed to get container status \"30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80\": rpc error: code = NotFound desc = could not find container \"30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80\": container with ID starting with 30d39b5c1636c1601970e100e171d3cd1c05ff4c56f9fb7342e409f123e83c80 not found: ID does not exist" Nov 02 22:01:19.913523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.913441 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk"] Nov 02 22:01:19.962199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:19.962160 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-34089f5b-openshift-default-68c7cb699f-4rjjk"] Nov 02 22:01:20.122063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:20.122027 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:20.331645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:20.331613 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:20.661027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:20.660939 4285 patch_prober.go:29] interesting pod/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:01:20.661027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:20.661006 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:21.176067 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.176029 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" path="/var/lib/kubelet/pods/db124196-7f89-46f5-a6cc-21782b23a2bc/volumes" Nov 02 22:01:21.523322 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.518030 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 22:01:21.541008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.540981 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 22:01:21.604915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.604821 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.604915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.604862 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.604915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.604893 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.604928 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.604968 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605036 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605107 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605153 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605560 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605580 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.605652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605617 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605694 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605791 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605828 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605858 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605947 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605983 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.606099 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.606161 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token\") pod \"12b7c971-1048-4f55-9bd3-d8f705f26f80\" (UID: \"12b7c971-1048-4f55-9bd3-d8f705f26f80\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.606210 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs\") pod \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\" (UID: \"08ab0a7c-02dc-4187-8d69-5f4fe48cb164\") " Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605692 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.605968 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.606203 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.606554 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.606861 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.607013 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.608616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.607293 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.609513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.609382 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.609513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.609430 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.609513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.609451 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.609513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.609471 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.610334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.610005 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.610334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.610275 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "kube-api-access-kl6kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.610689 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.610640 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:21.611456 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.611431 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.611532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.611498 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.612037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.611781 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m" pod "12b7c971-1048-4f55-9bd3-d8f705f26f80" (UID: "12b7c971-1048-4f55-9bd3-d8f705f26f80"). InnerVolumeSpecName "kube-api-access-kdt5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.616611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.616580 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo" pod "08ab0a7c-02dc-4187-8d69-5f4fe48cb164" (UID: "08ab0a7c-02dc-4187-8d69-5f4fe48cb164"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:21.618588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.618565 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 22:01:21.708387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.708353 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 22:01:21.710901 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.710866 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.710963 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711068 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711151 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711281 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711210 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711340 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711405 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711467 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.711584 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711564 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711569 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.711659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711607 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token\") pod \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\" (UID: \"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87\") " Nov 02 22:01:21.711773 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.711688 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.712047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.712021 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713057 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713088 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl6kz\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-kube-api-access-kl6kz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713105 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713119 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713157 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713172 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713185 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713198 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713218 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713232 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b7c971-1048-4f55-9bd3-d8f705f26f80-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713246 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713259 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713273 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713289 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713301 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdt5m\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-kube-api-access-kdt5m\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713315 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713329 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b7c971-1048-4f55-9bd3-d8f705f26f80-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713341 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ab0a7c-02dc-4187-8d69-5f4fe48cb164-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713356 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.713369 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.714214 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.724102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.714986 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.736458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.729281 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:21.736458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.729293 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.736458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.731499 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x" pod "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" (UID: "681cba6a-9e23-47a6-9aef-f2ffbd3fbd87"). InnerVolumeSpecName "kube-api-access-kp45x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.774534 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.774498 4285 generic.go:356] "Generic (PLEG): container finished" podID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerID="7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc" exitCode=0 Nov 02 22:01:21.774703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.774585 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" event={"ID":"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87","Type":"ContainerDied","Data":"7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc"} Nov 02 22:01:21.774703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.774665 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" event={"ID":"681cba6a-9e23-47a6-9aef-f2ffbd3fbd87","Type":"ContainerDied","Data":"1e11d2fcc4b221f4c44d5422d969ec1a5db4806fa1f088c76020d79c93e17b08"} Nov 02 22:01:21.774703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.774696 4285 scope.go:122] "RemoveContainer" containerID="7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc" Nov 02 22:01:21.774906 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.774894 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7" Nov 02 22:01:21.778846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.778712 4285 generic.go:356] "Generic (PLEG): container finished" podID="65c76022-a683-4360-9d08-24234e95257d" containerID="8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee" exitCode=0 Nov 02 22:01:21.778846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.778742 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" event={"ID":"65c76022-a683-4360-9d08-24234e95257d","Type":"ContainerDied","Data":"8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee"} Nov 02 22:01:21.779392 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.779347 4285 scope.go:122] "RemoveContainer" containerID="8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee" Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781382 4285 generic.go:356] "Generic (PLEG): container finished" podID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerID="ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b" exitCode=0 Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781550 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781591 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781614 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781648 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781719 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" event={"ID":"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd","Type":"ContainerDied","Data":"ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b"} Nov 02 22:01:21.781920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.781756 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh" event={"ID":"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd","Type":"ContainerDied","Data":"63a77c329a6dad5a91f70eab9b2fde9845152018034a8b5c88e43320c31a0d81"} Nov 02 22:01:21.784071 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.784045 4285 generic.go:356] "Generic (PLEG): container finished" podID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerID="ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224" exitCode=0 Nov 02 22:01:21.784214 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.784163 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" event={"ID":"12b7c971-1048-4f55-9bd3-d8f705f26f80","Type":"ContainerDied","Data":"ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224"} Nov 02 22:01:21.784214 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.784179 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" Nov 02 22:01:21.784343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.784220 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4" event={"ID":"12b7c971-1048-4f55-9bd3-d8f705f26f80","Type":"ContainerDied","Data":"5eaaed47f72b5fd5a5dd324c61b418a93c222c77e12a9ce57e6cd74357e742c6"} Nov 02 22:01:21.787420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.787397 4285 generic.go:356] "Generic (PLEG): container finished" podID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerID="efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd" exitCode=0 Nov 02 22:01:21.787515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.787434 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" event={"ID":"08ab0a7c-02dc-4187-8d69-5f4fe48cb164","Type":"ContainerDied","Data":"efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd"} Nov 02 22:01:21.787515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.787482 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" Nov 02 22:01:21.787628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.787483 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx" event={"ID":"08ab0a7c-02dc-4187-8d69-5f4fe48cb164","Type":"ContainerDied","Data":"646f539871df381cd7227555e09ac9e0b245789d03bf95853832a68f03ee2d46"} Nov 02 22:01:21.790336 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.790316 4285 scope.go:122] "RemoveContainer" containerID="7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc" Nov 02 22:01:21.790655 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:21.790625 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc\": container with ID starting with 7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc not found: ID does not exist" containerID="7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc" Nov 02 22:01:21.790743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.790665 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc"} err="failed to get container status \"7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc\": rpc error: code = NotFound desc = could not find container \"7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc\": container with ID starting with 7d36dd104dc1ca018513c4abab37397c0ece368ca363c8d7bf405b799f93d8fc not found: ID does not exist" Nov 02 22:01:21.790743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.790688 4285 scope.go:122] "RemoveContainer" containerID="ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b" Nov 02 22:01:21.828206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828170 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828224 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828257 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828419 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828506 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828590 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828618 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828810 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828702 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.828810 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.828754 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo\") pod \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\" (UID: \"49e9a3e8-77ae-41ab-b9b2-44acdd0697bd\") " Nov 02 22:01:21.830531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.830425 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp45x\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-kube-api-access-kp45x\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.830531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.830464 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.830531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.830479 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.830531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.830493 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.830531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.830508 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.833723 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.833694 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:21.833962 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.833941 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.834323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.834303 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.834499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.834454 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.834634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.834614 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.839080 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.839058 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "kube-api-access-ltq24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.841707 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.841684 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:21.841889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.841861 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:21.843767 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.843733 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token" pod "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" (UID: "49e9a3e8-77ae-41ab-b9b2-44acdd0697bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:21.896295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.896269 4285 scope.go:122] "RemoveContainer" containerID="ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b" Nov 02 22:01:21.896673 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:21.896646 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b\": container with ID starting with ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b not found: ID does not exist" containerID="ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b" Nov 02 22:01:21.896793 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.896686 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b"} err="failed to get container status \"ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b\": rpc error: code = NotFound desc = could not find container \"ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b\": container with ID starting with ee401408db200cd4e3eb1f3b47473ec1b5b02a5b2a5250ea95c645b89b7f924b not found: ID does not exist" Nov 02 22:01:21.896793 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.896713 4285 scope.go:122] "RemoveContainer" containerID="ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224" Nov 02 22:01:21.911008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.910988 4285 scope.go:122] "RemoveContainer" containerID="ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224" Nov 02 22:01:21.912997 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:21.912971 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224\": container with ID starting with ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224 not found: ID does not exist" containerID="ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224" Nov 02 22:01:21.913112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.913007 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224"} err="failed to get container status \"ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224\": rpc error: code = NotFound desc = could not find container \"ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224\": container with ID starting with ff6d3a82ad9426c57e78151c0dd23ad7eb9e9bdd38d3b288feba6ff4bb1ac224 not found: ID does not exist" Nov 02 22:01:21.913112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.913028 4285 scope.go:122] "RemoveContainer" containerID="efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934375 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934410 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934430 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934447 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934464 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934482 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltq24\" (UniqueName: \"kubernetes.io/projected/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-kube-api-access-ltq24\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934501 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934514 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.934842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.934527 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:21.951292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.950949 4285 scope.go:122] "RemoveContainer" containerID="f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0" Nov 02 22:01:21.963069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.962982 4285 scope.go:122] "RemoveContainer" containerID="efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd" Nov 02 22:01:21.963979 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:21.963958 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd\": container with ID starting with efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd not found: ID does not exist" containerID="efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd" Nov 02 22:01:21.964075 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.963986 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd"} err="failed to get container status \"efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd\": rpc error: code = NotFound desc = could not find container \"efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd\": container with ID starting with efc4703adbea3e0f5519e3fca06e5b93f635c04e963fdb98ed945bf3cfb194bd not found: ID does not exist" Nov 02 22:01:21.964075 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.964003 4285 scope.go:122] "RemoveContainer" containerID="f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0" Nov 02 22:01:21.964317 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:21.964297 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0\": container with ID starting with f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0 not found: ID does not exist" containerID="f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0" Nov 02 22:01:21.964380 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:21.964322 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0"} err="failed to get container status \"f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0\": rpc error: code = NotFound desc = could not find container \"f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0\": container with ID starting with f27c7d900e74eae28e8a74fde23e915c000603f0fc92def022c02d77c75cf8d0 not found: ID does not exist" Nov 02 22:01:22.080254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.080218 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7"] Nov 02 22:01:22.086588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.086556 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-clfw7"] Nov 02 22:01:22.183554 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.183466 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx"] Nov 02 22:01:22.190167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.190094 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-vsgfx"] Nov 02 22:01:22.295848 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.295813 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4"] Nov 02 22:01:22.299778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.299751 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-fllh4"] Nov 02 22:01:22.664215 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.664183 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh"] Nov 02 22:01:22.673583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.673051 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10fdd5-openshift-default-ffbc4d897-7cnhh"] Nov 02 22:01:22.798762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:22.798728 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" event={"ID":"65c76022-a683-4360-9d08-24234e95257d","Type":"ContainerStarted","Data":"2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff"} Nov 02 22:01:23.037549 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.037516 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 22:01:23.038087 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.038061 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:23.038648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.038624 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:23.038755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.038700 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:23.235149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.234289 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" path="/var/lib/kubelet/pods/08ab0a7c-02dc-4187-8d69-5f4fe48cb164/volumes" Nov 02 22:01:23.251393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.245906 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" path="/var/lib/kubelet/pods/12b7c971-1048-4f55-9bd3-d8f705f26f80/volumes" Nov 02 22:01:23.251393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.246805 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" path="/var/lib/kubelet/pods/49e9a3e8-77ae-41ab-b9b2-44acdd0697bd/volumes" Nov 02 22:01:23.273152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.255461 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" path="/var/lib/kubelet/pods/681cba6a-9e23-47a6-9aef-f2ffbd3fbd87/volumes" Nov 02 22:01:23.348583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.348449 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:23.507263 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.507024 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:23.771148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.769326 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f"] Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782851 4285 cpu_manager.go:414] "containerMap: removing container" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782883 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782918 4285 cpu_manager.go:414] "containerMap: removing container" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782926 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782941 4285 cpu_manager.go:414] "containerMap: removing container" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782949 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.782992 4285 cpu_manager.go:414] "containerMap: removing container" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783000 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783015 4285 cpu_manager.go:414] "containerMap: removing container" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783023 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783238 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e9a3e8-77ae-41ab-b9b2-44acdd0697bd" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783254 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="681cba6a-9e23-47a6-9aef-f2ffbd3fbd87" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783319 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783332 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783343 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783356 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b7c971-1048-4f55-9bd3-d8f705f26f80" containerName="istio-proxy" Nov 02 22:01:23.786159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.783365 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" Nov 02 22:01:23.822809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.820542 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:01:23.822809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.820887 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:23.897145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.887631 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-fb0367\"" Nov 02 22:01:23.980332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.915549 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-fb0367-dockercfg-nklc4\"" Nov 02 22:01:23.980332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.915580 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f"] Nov 02 22:01:23.980332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.960511 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:23.980332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:23.960572 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9n4p\" (UniqueName: \"kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p\") pod \"payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:24.053430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.052924 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:24.053430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.052986 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:24.093477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.091472 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:24.093477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.091553 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9n4p\" (UniqueName: \"kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p\") pod \"payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:24.093477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.093376 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:24.240258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.227924 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj"] Nov 02 22:01:24.248815 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.248693 4285 cpu_manager.go:414] "containerMap: removing container" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" Nov 02 22:01:24.249050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.249032 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db124196-7f89-46f5-a6cc-21782b23a2bc" containerName="istio-proxy" Nov 02 22:01:24.249287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.249270 4285 cpu_manager.go:414] "containerMap: removing container" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" Nov 02 22:01:24.249511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.249495 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ab0a7c-02dc-4187-8d69-5f4fe48cb164" containerName="istio-proxy" Nov 02 22:01:24.311197 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.290158 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:24.472183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.424558 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.576020 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.568981 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj"] Nov 02 22:01:24.604752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.600302 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pvn\" (UniqueName: \"kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn\") pod \"payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.605989 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.605953 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.611446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.611420 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9n4p\" (UniqueName: \"kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p\") pod \"payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:24.656899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.656856 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:24.698431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.698393 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:24.705519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.705497 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:24.708198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.708172 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pvn\" (UniqueName: \"kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn\") pod \"payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.708294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.708277 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.709140 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.709104 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.767048 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.766985 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pvn\" (UniqueName: \"kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn\") pod \"payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:24.954166 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.954116 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:24.963685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:24.963663 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:25.101018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.100595 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:25.101018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.100655 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:25.127177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.125480 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:25.276578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.276541 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f"] Nov 02 22:01:25.288120 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.288043 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" event={"ID":"7058b902-fcd3-411b-af6b-d4d63b8a4cba","Type":"ContainerStarted","Data":"e30d478665df53c9fa3719a8a87a09bbbe2ca52e5a87959ec919bd83702505a2"} Nov 02 22:01:25.581649 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:01:25.581616 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a92a95_a4bf_4805_a8ce_e2b0c0206c87.slice/crio-98bd75f299a68b2eabaae53a95c6f1d2ffa86fe27abf4596b938f159c8bef2c9 WatchSource:0}: Error finding container 98bd75f299a68b2eabaae53a95c6f1d2ffa86fe27abf4596b938f159c8bef2c9: Status 404 returned error can't find the container with id 98bd75f299a68b2eabaae53a95c6f1d2ffa86fe27abf4596b938f159c8bef2c9 Nov 02 22:01:25.585120 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.585084 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj"] Nov 02 22:01:25.730156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:25.729229 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 22:01:26.038586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.038543 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:26.039194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.039159 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:26.298458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.298365 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" event={"ID":"7058b902-fcd3-411b-af6b-d4d63b8a4cba","Type":"ContainerStarted","Data":"61ce2074b27e48297c8d4d8fb940083dbdadc9f326d6822526ffb18300f7685a"} Nov 02 22:01:26.298458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.298451 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:26.300164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.300135 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" event={"ID":"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87","Type":"ContainerStarted","Data":"314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222"} Nov 02 22:01:26.300300 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.300180 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" event={"ID":"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87","Type":"ContainerStarted","Data":"98bd75f299a68b2eabaae53a95c6f1d2ffa86fe27abf4596b938f159c8bef2c9"} Nov 02 22:01:26.300300 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.300201 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:26.448192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.446526 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" podStartSLOduration=3.4465035459999998 podStartE2EDuration="3.446503546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:26.410440103 +0000 UTC m=+1625.371990424" watchObservedRunningTime="2026-11-02 22:01:26.446503546 +0000 UTC m=+1625.408053868" Nov 02 22:01:26.547147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.546293 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" podStartSLOduration=3.546265567 podStartE2EDuration="3.546265567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:26.527005606 +0000 UTC m=+1625.488556120" watchObservedRunningTime="2026-11-02 22:01:26.546265567 +0000 UTC m=+1625.507815911" Nov 02 22:01:26.708154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.705295 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:26.898261 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.898221 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269"] Nov 02 22:01:26.909165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:26.898785 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" containerID="cri-o://97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2" gracePeriod=30 Nov 02 22:01:27.055349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:27.055249 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:27.055349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:27.055317 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:27.449153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:27.449053 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:28.039002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.036575 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:28.039002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.036641 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:28.460987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.460951 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn"] Nov 02 22:01:28.465966 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.465942 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.555721 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.555678 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-204000-openshift-default-dockercfg-jw8gg\"" Nov 02 22:01:28.569640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.569560 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:28.585389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.584861 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.589802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.589710 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.590115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.590089 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.590484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.590460 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.590679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.590661 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.590871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.590851 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.591181 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.591162 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.591401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.591381 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.591583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.591563 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblzv\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.757729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.757637 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.764324 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.764645 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.764815 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.764900 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.764969 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.765104 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.765193 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.765243 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblzv\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.766315 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.767148 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.767436 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.774564 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn"] Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.774926 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.801179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.780535 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.892952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.892653 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.892952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.892871 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblzv\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.908339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.901351 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:28.908339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.907812 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token\") pod \"e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:29.001147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:28.993791 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:29.044198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.043597 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 02 22:01:29.044198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.043665 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 02 22:01:29.081485 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.081453 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:29.533218 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:01:29.533186 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedaaab34_eddb_4c62_b119_b51fa28512c9.slice/crio-12c503567e3e3190d974ce7ccf279e3b35a3e2de4d347d3da6f397b37884483d WatchSource:0}: Error finding container 12c503567e3e3190d974ce7ccf279e3b35a3e2de4d347d3da6f397b37884483d: Status 404 returned error can't find the container with id 12c503567e3e3190d974ce7ccf279e3b35a3e2de4d347d3da6f397b37884483d Nov 02 22:01:29.533654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.533246 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn"] Nov 02 22:01:29.536817 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.536784 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:29.536946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.536847 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:29.536946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:29.536877 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:30.017371 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.017338 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:30.040990 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.040958 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 22:01:30.042553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.042525 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 22:01:30.201155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.200787 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:30.603634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.603204 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" event={"ID":"edaaab34-eddb-4c62-b119-b51fa28512c9","Type":"ContainerStarted","Data":"c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def"} Nov 02 22:01:30.603634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.603265 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" event={"ID":"edaaab34-eddb-4c62-b119-b51fa28512c9","Type":"ContainerStarted","Data":"12c503567e3e3190d974ce7ccf279e3b35a3e2de4d347d3da6f397b37884483d"} Nov 02 22:01:30.724560 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:30.724510 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podStartSLOduration=2.7244976640000003 podStartE2EDuration="2.724497664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:30.721912733 +0000 UTC m=+1629.683463054" watchObservedRunningTime="2026-11-02 22:01:30.724497664 +0000 UTC m=+1629.686047983" Nov 02 22:01:31.082278 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:31.082242 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:31.087637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:31.087613 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:31.609742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:31.609714 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:31.610612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:31.610591 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:01:32.375971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.375945 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 22:01:32.436984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.436950 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.437196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437045 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.437196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437107 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.437196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437185 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437353 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437399 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437432 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437545 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437589 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data\") pod \"ea7abda0-213a-41bb-be21-4f6916d45cfe\" (UID: \"ea7abda0-213a-41bb-be21-4f6916d45cfe\") " Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437626 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.437741 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.438380 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.439427 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.440607 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.441470 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "kube-api-access-ws4rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.441877 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.441904 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7abda0-213a-41bb-be21-4f6916d45cfe-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.441920 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.441949 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.442204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.441966 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.443164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.443100 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:32.443262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.443168 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:32.443262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.443211 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token" pod "ea7abda0-213a-41bb-be21-4f6916d45cfe" (UID: "ea7abda0-213a-41bb-be21-4f6916d45cfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:32.544256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.544225 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.544256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.544255 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws4rd\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-kube-api-access-ws4rd\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.544451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.544271 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.544451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.544280 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7abda0-213a-41bb-be21-4f6916d45cfe-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:01:32.622493 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.622405 4285 generic.go:356] "Generic (PLEG): container finished" podID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerID="97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2" exitCode=0 Nov 02 22:01:32.623294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.623273 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" Nov 02 22:01:32.624487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.624420 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" event={"ID":"ea7abda0-213a-41bb-be21-4f6916d45cfe","Type":"ContainerDied","Data":"97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2"} Nov 02 22:01:32.624670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.624543 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269" event={"ID":"ea7abda0-213a-41bb-be21-4f6916d45cfe","Type":"ContainerDied","Data":"1f2caccfb91f1585e663fa951f787d09993597cb0d9ba6a32887b4a1f64ad4a3"} Nov 02 22:01:32.624670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.624577 4285 scope.go:122] "RemoveContainer" containerID="97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2" Nov 02 22:01:32.638575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.638556 4285 scope.go:122] "RemoveContainer" containerID="02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb" Nov 02 22:01:32.648306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.648288 4285 scope.go:122] "RemoveContainer" containerID="97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2" Nov 02 22:01:32.648615 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:32.648595 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2\": container with ID starting with 97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2 not found: ID does not exist" containerID="97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2" Nov 02 22:01:32.648706 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.648621 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2"} err="failed to get container status \"97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2\": rpc error: code = NotFound desc = could not find container \"97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2\": container with ID starting with 97c86120caef184c3f2cd33aac888610ab9d73fccb6a9ee12404ce939dc440e2 not found: ID does not exist" Nov 02 22:01:32.648706 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.648640 4285 scope.go:122] "RemoveContainer" containerID="02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb" Nov 02 22:01:32.648924 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:32.648910 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb\": container with ID starting with 02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb not found: ID does not exist" containerID="02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb" Nov 02 22:01:32.648981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.648929 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb"} err="failed to get container status \"02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb\": rpc error: code = NotFound desc = could not find container \"02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb\": container with ID starting with 02c1b587d0a0a76cb37566ebea70ddd3cf12dde3571d198f6d9fdd8741cb41cb not found: ID does not exist" Nov 02 22:01:32.790328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.790295 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269"] Nov 02 22:01:32.795755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:32.795724 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a4e90e53-openshift-default-854fb84545-xc269"] Nov 02 22:01:33.175329 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:33.175297 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" path="/var/lib/kubelet/pods/ea7abda0-213a-41bb-be21-4f6916d45cfe/volumes" Nov 02 22:01:33.476141 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:33.476104 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:33.646643 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:33.646612 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:33.812226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:33.812121 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:35.262236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:35.262184 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:37.327404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:37.327375 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:01:37.327957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:37.327939 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:01:38.895636 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:38.895598 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:40.577206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.576225 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 22:01:40.577803 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.577774 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:40.679270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.679220 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:40.715266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.715216 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 22:01:40.756259 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.756207 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:40.795149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.793214 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:40.824252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.824206 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:40.973633 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:40.973584 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:41.443518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.443277 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.545870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.545824 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.656455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.577232 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.682177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.580212 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.682177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.664322 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 22:01:41.682177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.664414 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 22:01:41.682177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.680084 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.682177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.680842 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 22:01:41.687913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.687882 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:41.751202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.735061 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.751889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.751862 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 22:01:41.794213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.766237 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.794213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.766854 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 22:01:41.794984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.794924 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.906145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.898309 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:41.906145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:41.898582 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 22:01:42.013393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.013295 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:42.079704 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.059097 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:42.214146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.207724 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:42.421199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.420767 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj"] Nov 02 22:01:42.431157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.421966 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" Nov 02 22:01:42.431157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.421996 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" Nov 02 22:01:42.431157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.422017 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" Nov 02 22:01:42.431157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.422025 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" Nov 02 22:01:42.431157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.422240 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" Nov 02 22:01:42.431157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.422263 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7abda0-213a-41bb-be21-4f6916d45cfe" containerName="istio-proxy" Nov 02 22:01:42.538151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.504323 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:42.579689 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.579655 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-204000\"" Nov 02 22:01:42.589147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.580158 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-204000-dockercfg-dpg8x\"" Nov 02 22:01:42.692752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.692620 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 22:01:42.694159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.694106 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 22:01:42.714434 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.694919 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:42.809579 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.692490 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:42.809880 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.752052 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 22:01:42.834389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.773256 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:42.834934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.801572 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:42.835423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.835402 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 22:01:42.870085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.870047 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:42.913198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.903598 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vhz\" (UniqueName: \"kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz\") pod \"payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:43.021736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:42.934203 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:43.048489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.048456 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj"] Nov 02 22:01:43.065566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.065525 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x"] Nov 02 22:01:43.145162 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.131367 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vhz\" (UniqueName: \"kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz\") pod \"payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:43.153160 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.145612 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:43.219149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.193944 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:43.349241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.349148 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:43.606232 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.589270 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:43.634154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.623884 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vhz\" (UniqueName: \"kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz\") pod \"payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:43.634154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.624398 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:43.662154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.658484 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z826g\" (UniqueName: \"kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g\") pod \"payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:43.662154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.658742 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:43.724153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.701075 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:43.725171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.725110 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 22:01:43.839366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.839305 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:43.865884 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.865791 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 02 22:01:43.979404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.871327 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 02 22:01:43.979404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.883408 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:43.979404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.895342 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:43.979404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.978599 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z826g\" (UniqueName: \"kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g\") pod \"payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:43.979404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.932823 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:43.979790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:43.896286 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:44.081463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.066205 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 22:01:44.099396 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.099362 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x"] Nov 02 22:01:44.099692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.099677 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 22:01:44.123213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.114858 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z826g\" (UniqueName: \"kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g\") pod \"payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:44.287148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.258117 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 22:01:44.383388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.350475 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:44.572169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.569355 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 22:01:44.771177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.761972 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 02 22:01:44.802161 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.800732 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 22:01:44.900604 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.900520 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:01:45.002150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.998278 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 22:01:45.002150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:44.998757 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:45.004472 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.004448 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 22:01:45.004669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.004658 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 22:01:45.094207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.093639 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 22:01:45.104337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.103419 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 22:01:45.106420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.104770 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv"] Nov 02 22:01:45.110499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.110219 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.111630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.111609 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv"] Nov 02 22:01:45.209091 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.208905 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2be7cc3\"/\"default-dockercfg-fdg9f\"" Nov 02 22:01:45.209588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.209566 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2be7cc3\"/\"e2e-lifecycle-model-e2be7cc3-kserve-self-signed-certs\"" Nov 02 22:01:45.245150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.212972 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.298152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.293513 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzbjv\" (UniqueName: \"kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.298152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.293609 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.298152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.293950 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.298152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.294395 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.298152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.294458 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.344505 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.271195 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2be7cc3\"/\"kube-root-ca.crt\"" Nov 02 22:01:45.344505 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.271616 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2be7cc3\"/\"openshift-service-ca.crt\"" Nov 02 22:01:45.458148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.412851 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzbjv\" (UniqueName: \"kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.608221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.608084 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.608588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.608536 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.609054 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.609027 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.609243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.609216 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.609585 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.609556 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.610209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.610179 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.610890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.610635 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.621939 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.621790 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.736342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.706786 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzbjv\" (UniqueName: \"kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.736342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.707355 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.736342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.711166 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj"] Nov 02 22:01:45.758057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.740480 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm\") pod \"e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:45.909898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.909794 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" event={"ID":"9b4043e2-d29b-44e4-844b-5fc0a2d9201a","Type":"ContainerStarted","Data":"4acfd40a7439e58331babf2c7404378342d17c871fdc977b752468738f476cab"} Nov 02 22:01:46.000328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:45.999280 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:46.008523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.008496 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:46.253159 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:01:46.237408 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:01:46.503741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.499831 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x"] Nov 02 22:01:46.927267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.925936 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" event={"ID":"9b4043e2-d29b-44e4-844b-5fc0a2d9201a","Type":"ContainerStarted","Data":"ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248"} Nov 02 22:01:46.927267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.927207 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:46.930694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.929764 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" event={"ID":"6aa9aecd-6b68-459f-82f5-a32118a08a0b","Type":"ContainerStarted","Data":"5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99"} Nov 02 22:01:46.930694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.929817 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" event={"ID":"6aa9aecd-6b68-459f-82f5-a32118a08a0b","Type":"ContainerStarted","Data":"b5d80b4d9608fadf114067cbf80b4400c371ebdd3ea5d932d85e6438628ee16e"} Nov 02 22:01:46.930694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.930643 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:46.965009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:46.952662 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv"] Nov 02 22:01:47.173249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:47.162225 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" podStartSLOduration=5.162205769 podStartE2EDuration="5.162205769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:47.046696742 +0000 UTC m=+1646.008247066" watchObservedRunningTime="2026-11-02 22:01:47.162205769 +0000 UTC m=+1646.123756092" Nov 02 22:01:47.194633 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:47.193843 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" podStartSLOduration=5.193821432 podStartE2EDuration="5.193821432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:47.160541846 +0000 UTC m=+1646.122092166" watchObservedRunningTime="2026-11-02 22:01:47.193821432 +0000 UTC m=+1646.155371753" Nov 02 22:01:47.941541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:47.941445 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" event={"ID":"af54beaf-c72f-40c6-adaa-a122e0d0301b","Type":"ContainerStarted","Data":"a166fb59034da455625e1c7b28d2ec570178db14b6168f8de138506b2e40ad62"} Nov 02 22:01:47.941541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:47.941499 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" event={"ID":"af54beaf-c72f-40c6-adaa-a122e0d0301b","Type":"ContainerStarted","Data":"e3cd222fa0bb2db9814eedbad6ba06903d6e99bc0889b259df64713207540b68"} Nov 02 22:01:48.000284 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:48.000255 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:48.058872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:48.058813 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" podStartSLOduration=5.058795004 podStartE2EDuration="5.058795004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:48.054581233 +0000 UTC m=+1647.016131556" watchObservedRunningTime="2026-11-02 22:01:48.058795004 +0000 UTC m=+1647.020345326" Nov 02 22:01:50.764146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:50.762703 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:51.980438 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.980407 4285 generic.go:356] "Generic (PLEG): container finished" podID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerID="4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b" exitCode=0 Nov 02 22:01:51.980965 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.980475 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" event={"ID":"9a87d989-eba7-430a-ac2c-1c45fe0cef7a","Type":"ContainerDied","Data":"4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b"} Nov 02 22:01:51.981061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.981044 4285 scope.go:122] "RemoveContainer" containerID="4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b" Nov 02 22:01:51.982320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.982302 4285 generic.go:356] "Generic (PLEG): container finished" podID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerID="9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3" exitCode=0 Nov 02 22:01:51.982398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.982374 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" event={"ID":"b7f04c3b-044c-40a2-a010-cc06273a9103","Type":"ContainerDied","Data":"9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3"} Nov 02 22:01:51.982805 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.982790 4285 scope.go:122] "RemoveContainer" containerID="9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3" Nov 02 22:01:51.982995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.982969 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:51.983050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.983037 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:51.983104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.983067 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:51.984086 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.984061 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:51.984225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.984119 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:51.984225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:51.984188 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:01:52.992362 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:52.992326 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" event={"ID":"9a87d989-eba7-430a-ac2c-1c45fe0cef7a","Type":"ContainerStarted","Data":"7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315"} Nov 02 22:01:52.994853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:52.994823 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" event={"ID":"b7f04c3b-044c-40a2-a010-cc06273a9103","Type":"ContainerStarted","Data":"9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1"} Nov 02 22:01:53.297211 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:53.297091 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:01:53.298169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:53.298145 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 22:01:53.298301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:53.298188 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 22:01:53.379184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:53.376184 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 22:01:53.379184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:53.378327 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:01:53.379184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:53.378399 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:01:54.297406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:54.297373 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 22:01:54.297820 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:54.297434 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 22:01:54.374630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:54.374587 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:01:54.374913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:54.374882 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:01:55.297250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:55.297213 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 22:01:55.297441 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:55.297270 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 22:01:55.373832 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:55.373657 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:01:55.373832 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:55.373733 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:01:55.380107 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:55.380068 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:55.999862 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:55.999823 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:56.064148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:56.059479 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:56.085284 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:56.085253 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:01:56.296981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:56.296888 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 22:01:56.296981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:56.296960 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 22:01:56.377154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:56.376307 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:01:56.377154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:56.376398 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:01:57.291627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:57.291593 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 22:01:57.296349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:57.296323 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 22:01:57.296446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:57.296381 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 22:01:57.373880 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:57.373838 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:01:57.374054 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:57.373892 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:01:57.904828 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:57.904796 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:01:58.296590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.296554 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 22:01:58.296761 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.296621 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 22:01:58.374388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.374354 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:01:58.374602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.374415 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 02 22:01:58.426830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.426784 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:58.956525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.956496 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:01:58.956930 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:58.956574 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:01:59.301314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:59.301242 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:01:59.302401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:59.302378 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:01:59.922167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:59.922118 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 22:01:59.925176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:01:59.925150 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 22:02:00.114369 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.114338 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:00.599189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.599115 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6"] Nov 02 22:02:00.599478 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.599448 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" containerID="cri-o://aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8" gracePeriod=30 Nov 02 22:02:00.603679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.603655 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw"] Nov 02 22:02:00.603847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.603828 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" containerID="cri-o://cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215" gracePeriod=30 Nov 02 22:02:00.649792 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.649759 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5"] Nov 02 22:02:00.650025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.649997 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" containerID="cri-o://547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66" gracePeriod=30 Nov 02 22:02:00.654801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.654773 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w"] Nov 02 22:02:00.655005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:00.654983 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" containerID="cri-o://89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0" gracePeriod=30 Nov 02 22:02:01.574464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:01.574428 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:02.012830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.012777 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:02.821847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.821813 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h"] Nov 02 22:02:02.828180 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.828149 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.900096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.900059 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-75577e-openshift-default-dockercfg-bkd25\"" Nov 02 22:02:02.934367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.934289 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.934645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.934623 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.934781 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.934762 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.935057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.935036 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.935245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.935226 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.935384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.935368 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.935531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.935512 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.935677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.935659 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.935808 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.935792 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8b4s\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:02.959066 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.959024 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h"] Nov 02 22:02:02.984821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:02.984783 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:03.137974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.137880 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.138652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.138629 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.139206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.139184 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.139834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.139814 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.205916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.205878 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.205934 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.205992 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.206062 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.206113 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8b4s\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.206534 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.139585 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.206802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.175803 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.207012 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.138553 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.207208 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.207119 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.207613 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.207582 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.210587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.210564 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.238894 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.238860 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:03.262742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.262705 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.262883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.262760 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8b4s\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s\") pod \"e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:03.398606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.398520 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:03.540328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:03.540290 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:04.075220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.075190 4285 patch_prober.go:29] interesting pod/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:02:04.075755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.075729 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:02:04.236485 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.236450 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr"] Nov 02 22:02:04.241531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.241505 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.285148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.285087 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.285511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.285484 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.285690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.285665 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.285801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.285706 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.286007 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.285982 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.286181 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.286163 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.286270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.286195 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6v9\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.286407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.286367 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.286628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.286608 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.291031 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.291005 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr"] Nov 02 22:02:04.293233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.293202 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4fc235-openshift-default-dockercfg-phk9w\"" Nov 02 22:02:04.319171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.319138 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:04.380884 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:04.380842 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbf506d_f704_4229_9d75_e2be5bd77925.slice/crio-cdea43179a94c3e7f436f4dc85979a9b76606d65b9995fa6051a681b128eb6f6 WatchSource:0}: Error finding container cdea43179a94c3e7f436f4dc85979a9b76606d65b9995fa6051a681b128eb6f6: Status 404 returned error can't find the container with id cdea43179a94c3e7f436f4dc85979a9b76606d65b9995fa6051a681b128eb6f6 Nov 02 22:02:04.381178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.381156 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h"] Nov 02 22:02:04.383742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.383713 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:04.383852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.383774 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:04.383852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.383801 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:04.388035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388013 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388048 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388078 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388175 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388314 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388351 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6v9\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388461 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388473 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388780 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.388910 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.388846 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.389438 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.389411 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.389438 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.389427 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.389562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.389113 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.389633 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.389611 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.391474 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.391450 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.392003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.391977 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.466144 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.452345 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6v9\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.490305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.488973 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token\") pod \"e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.496785 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.496734 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:04.569147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.561457 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:04.993508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:04.993472 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:05.107149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.102242 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr"] Nov 02 22:02:05.143663 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.128098 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:05.143999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.143967 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:05.144218 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.144194 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:05.234027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.233994 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" event={"ID":"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10","Type":"ContainerStarted","Data":"0fdf3c09d3b2a0854bf7e86628a2614f5a60b16907ca3b93077c8ec1b29110de"} Nov 02 22:02:05.236513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.235808 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" event={"ID":"6dbf506d-f704-4229-9d75-e2be5bd77925","Type":"ContainerStarted","Data":"4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec"} Nov 02 22:02:05.236513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.235848 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" event={"ID":"6dbf506d-f704-4229-9d75-e2be5bd77925","Type":"ContainerStarted","Data":"cdea43179a94c3e7f436f4dc85979a9b76606d65b9995fa6051a681b128eb6f6"} Nov 02 22:02:05.357314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.357250 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" podStartSLOduration=3.357229908 podStartE2EDuration="3.357229908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:05.350019963 +0000 UTC m=+1664.311570285" watchObservedRunningTime="2026-11-02 22:02:05.357229908 +0000 UTC m=+1664.318780230" Nov 02 22:02:05.541290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.541201 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:05.564172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.562603 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:05.996250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:05.994262 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 22:02:06.052974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.052850 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 22:02:06.054225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.054113 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063256 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063320 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063387 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063448 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063493 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063599 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063666 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063740 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.064934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.063928 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo\") pod \"c3662e24-62ab-48c7-86ae-63a96df54380\" (UID: \"c3662e24-62ab-48c7-86ae-63a96df54380\") " Nov 02 22:02:06.066620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.066300 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.066620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.066471 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.066620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.066530 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.066620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.066601 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.067085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.067013 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "kube-api-access-mbh7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.067612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.067581 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:06.067691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.067630 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:06.069766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.069715 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.069924 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.069882 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token" pod "c3662e24-62ab-48c7-86ae-63a96df54380" (UID: "c3662e24-62ab-48c7-86ae-63a96df54380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.076765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.076749 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 22:02:06.165553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165512 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165598 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165653 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165705 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165747 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165810 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165845 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165893 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165949 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.165992 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166054 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166177 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166226 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166305 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166332 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166364 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166406 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166446 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166558 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166601 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166635 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data\") pod \"073a27af-aa00-4982-91b0-9509297881f6\" (UID: \"073a27af-aa00-4982-91b0-9509297881f6\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166683 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166722 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166761 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166814 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166859 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f\") pod \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\" (UID: \"ed2f2a53-073d-4a89-98cc-9ccd2618b734\") " Nov 02 22:02:06.166935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.166903 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket\") pod \"298aee9e-ab9c-42af-a62c-ef8d48719bff\" (UID: \"298aee9e-ab9c-42af-a62c-ef8d48719bff\") " Nov 02 22:02:06.167936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.167908 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3662e24-62ab-48c7-86ae-63a96df54380-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.167936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.167944 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3662e24-62ab-48c7-86ae-63a96df54380-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.167960 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.167977 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.167992 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbh7p\" (UniqueName: \"kubernetes.io/projected/c3662e24-62ab-48c7-86ae-63a96df54380-kube-api-access-mbh7p\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168007 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168023 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168037 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168051 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3662e24-62ab-48c7-86ae-63a96df54380-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.168143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168112 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.169248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168705 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.169248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.168914 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.169248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.169134 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.169487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.169342 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:06.169487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.169382 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.169611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.169556 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.170390 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.170362 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:06.171272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.170964 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.171272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.171209 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "kube-api-access-cdss8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.171428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.171305 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.171558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.171539 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.171739 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.171718 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.172034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.172012 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:06.172348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.172325 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.172528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.172507 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.172986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.172917 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.173419 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.173387 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "kube-api-access-pcl25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.174411 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.174388 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.176591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.176379 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:06.177684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.177646 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "kube-api-access-6fz6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.181612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.181441 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:06.181612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.181452 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.181612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.181466 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:06.181612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.181581 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token" pod "ed2f2a53-073d-4a89-98cc-9ccd2618b734" (UID: "ed2f2a53-073d-4a89-98cc-9ccd2618b734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:06.183268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.183108 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy" pod "073a27af-aa00-4982-91b0-9509297881f6" (UID: "073a27af-aa00-4982-91b0-9509297881f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.183268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.183251 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs" pod "298aee9e-ab9c-42af-a62c-ef8d48719bff" (UID: "298aee9e-ab9c-42af-a62c-ef8d48719bff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:06.250045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.250011 4285 generic.go:356] "Generic (PLEG): container finished" podID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerID="89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0" exitCode=0 Nov 02 22:02:06.250455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.250425 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" event={"ID":"ed2f2a53-073d-4a89-98cc-9ccd2618b734","Type":"ContainerDied","Data":"89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0"} Nov 02 22:02:06.250638 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.250618 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" event={"ID":"ed2f2a53-073d-4a89-98cc-9ccd2618b734","Type":"ContainerDied","Data":"8360c4ad3a87ff0347a8f5c3e94064d7a0e775a38c4770c4fb147d117d4b7ddf"} Nov 02 22:02:06.254202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.250773 4285 scope.go:122] "RemoveContainer" containerID="89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0" Nov 02 22:02:06.254202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.250957 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w" Nov 02 22:02:06.256209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.256180 4285 generic.go:356] "Generic (PLEG): container finished" podID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerID="547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66" exitCode=0 Nov 02 22:02:06.256382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.256317 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" event={"ID":"298aee9e-ab9c-42af-a62c-ef8d48719bff","Type":"ContainerDied","Data":"547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66"} Nov 02 22:02:06.256382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.256356 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" event={"ID":"298aee9e-ab9c-42af-a62c-ef8d48719bff","Type":"ContainerDied","Data":"774d01f5d709ebf8cc5fd7600001804b031e4491e6012c00f1528ad7acf6a1b8"} Nov 02 22:02:06.257324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.256913 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5" Nov 02 22:02:06.258391 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.258367 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" event={"ID":"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10","Type":"ContainerStarted","Data":"4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4"} Nov 02 22:02:06.267163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.264242 4285 generic.go:356] "Generic (PLEG): container finished" podID="073a27af-aa00-4982-91b0-9509297881f6" containerID="aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8" exitCode=0 Nov 02 22:02:06.267163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.264319 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" event={"ID":"073a27af-aa00-4982-91b0-9509297881f6","Type":"ContainerDied","Data":"aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8"} Nov 02 22:02:06.267163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.264354 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" event={"ID":"073a27af-aa00-4982-91b0-9509297881f6","Type":"ContainerDied","Data":"2acaf3982573e11e0e3c68668b6fd81e7e63b97b8112ec322055f6d0f74760be"} Nov 02 22:02:06.267163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.264422 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6" Nov 02 22:02:06.270450 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.270039 4285 scope.go:122] "RemoveContainer" containerID="89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271416 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcl25\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-kube-api-access-pcl25\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271441 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271456 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdss8\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-kube-api-access-cdss8\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271473 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271488 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:06.271493 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0\": container with ID starting with 89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0 not found: ID does not exist" containerID="89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271535 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0"} err="failed to get container status \"89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0\": rpc error: code = NotFound desc = could not find container \"89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0\": container with ID starting with 89ec6b75db09874561e6232e1de2a9eb86b4499f6daf7f944d4e19fb1b8f1df0 not found: ID does not exist" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271558 4285 scope.go:122] "RemoveContainer" containerID="547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271504 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271635 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271652 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271666 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073a27af-aa00-4982-91b0-9509297881f6-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271680 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271695 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271710 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073a27af-aa00-4982-91b0-9509297881f6-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271723 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271736 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271750 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271766 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298aee9e-ab9c-42af-a62c-ef8d48719bff-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271779 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271793 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271808 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073a27af-aa00-4982-91b0-9509297881f6-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271822 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298aee9e-ab9c-42af-a62c-ef8d48719bff-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271836 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271849 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2f2a53-073d-4a89-98cc-9ccd2618b734-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271863 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2f2a53-073d-4a89-98cc-9ccd2618b734-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271877 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fz6f\" (UniqueName: \"kubernetes.io/projected/ed2f2a53-073d-4a89-98cc-9ccd2618b734-kube-api-access-6fz6f\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271892 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271906 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073a27af-aa00-4982-91b0-9509297881f6-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.272136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.271920 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298aee9e-ab9c-42af-a62c-ef8d48719bff-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:06.274986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.274965 4285 generic.go:356] "Generic (PLEG): container finished" podID="c3662e24-62ab-48c7-86ae-63a96df54380" containerID="cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215" exitCode=0 Nov 02 22:02:06.275960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.275392 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" Nov 02 22:02:06.275960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.275497 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" event={"ID":"c3662e24-62ab-48c7-86ae-63a96df54380","Type":"ContainerDied","Data":"cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215"} Nov 02 22:02:06.275960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.275531 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw" event={"ID":"c3662e24-62ab-48c7-86ae-63a96df54380","Type":"ContainerDied","Data":"cca1f9535f98898e98921c9f02a18e0fd9af4bb39bb351fd59e209211f44300d"} Nov 02 22:02:06.275960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.275823 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:06.277017 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.276974 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:02:06.288108 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.288088 4285 scope.go:122] "RemoveContainer" containerID="547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66" Nov 02 22:02:06.288437 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:06.288407 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66\": container with ID starting with 547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66 not found: ID does not exist" containerID="547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66" Nov 02 22:02:06.288550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.288446 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66"} err="failed to get container status \"547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66\": rpc error: code = NotFound desc = could not find container \"547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66\": container with ID starting with 547e5ab8b00713c86bf5563d5317e7e930b7f7615973629b8b91f38f4497cb66 not found: ID does not exist" Nov 02 22:02:06.288550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.288469 4285 scope.go:122] "RemoveContainer" containerID="aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8" Nov 02 22:02:06.298428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.298405 4285 scope.go:122] "RemoveContainer" containerID="aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8" Nov 02 22:02:06.298717 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:06.298693 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8\": container with ID starting with aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8 not found: ID does not exist" containerID="aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8" Nov 02 22:02:06.298798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.298725 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8"} err="failed to get container status \"aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8\": rpc error: code = NotFound desc = could not find container \"aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8\": container with ID starting with aedada4ba2e07b819d72dcde3576fa4b622bde3a5acd14fbcea6846345a67cd8 not found: ID does not exist" Nov 02 22:02:06.298798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.298743 4285 scope.go:122] "RemoveContainer" containerID="cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215" Nov 02 22:02:06.307883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.307869 4285 scope.go:122] "RemoveContainer" containerID="cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215" Nov 02 22:02:06.308270 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:06.308215 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215\": container with ID starting with cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215 not found: ID does not exist" containerID="cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215" Nov 02 22:02:06.308270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.308248 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215"} err="failed to get container status \"cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215\": rpc error: code = NotFound desc = could not find container \"cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215\": container with ID starting with cd474063ab950a9bae6b1f4547aef460723569b150da28d2c39cd5e111b9b215 not found: ID does not exist" Nov 02 22:02:06.562769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.562740 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:06.568266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.568242 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:06.598150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.598104 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6"] Nov 02 22:02:06.602607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.602581 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-bczm6"] Nov 02 22:02:06.706360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.706314 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podStartSLOduration=2.706300764 podStartE2EDuration="2.706300764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:06.702351729 +0000 UTC m=+1665.663902061" watchObservedRunningTime="2026-11-02 22:02:06.706300764 +0000 UTC m=+1665.667851083" Nov 02 22:02:06.871199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.869462 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w"] Nov 02 22:02:06.899150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.898514 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-5lj4w"] Nov 02 22:02:06.982180 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.979335 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw"] Nov 02 22:02:06.994147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:06.992146 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-cqsmw"] Nov 02 22:02:07.080951 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.080922 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5"] Nov 02 22:02:07.084766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.084744 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9cc56b-openshift-default-6d75946c77-9qdw5"] Nov 02 22:02:07.175614 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.175544 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073a27af-aa00-4982-91b0-9509297881f6" path="/var/lib/kubelet/pods/073a27af-aa00-4982-91b0-9509297881f6/volumes" Nov 02 22:02:07.176090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.176075 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" path="/var/lib/kubelet/pods/298aee9e-ab9c-42af-a62c-ef8d48719bff/volumes" Nov 02 22:02:07.176575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.176561 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" path="/var/lib/kubelet/pods/c3662e24-62ab-48c7-86ae-63a96df54380/volumes" Nov 02 22:02:07.177393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.177378 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" path="/var/lib/kubelet/pods/ed2f2a53-073d-4a89-98cc-9ccd2618b734/volumes" Nov 02 22:02:07.286062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.286021 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:07.286903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:07.286882 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:02:08.145934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:08.145888 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:08.311027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:08.310992 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:08.479884 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:08.479855 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:09.961105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:09.961069 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:10.747023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:10.746990 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:10.940856 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:10.940821 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.105182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.105074 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.364528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.364430 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.592099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.583033 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.750156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.749700 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:11.892590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.892553 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq"] Nov 02 22:02:11.893948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.893917 4285 cpu_manager.go:414] "containerMap: removing container" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" Nov 02 22:02:11.894110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894096 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" Nov 02 22:02:11.894294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894282 4285 cpu_manager.go:414] "containerMap: removing container" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" Nov 02 22:02:11.894420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894409 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" Nov 02 22:02:11.894578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894568 4285 cpu_manager.go:414] "containerMap: removing container" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" Nov 02 22:02:11.894665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894654 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" Nov 02 22:02:11.894776 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894767 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" Nov 02 22:02:11.894854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.894845 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" Nov 02 22:02:11.895408 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.895386 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="073a27af-aa00-4982-91b0-9509297881f6" containerName="istio-proxy" Nov 02 22:02:11.895567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.895548 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2f2a53-073d-4a89-98cc-9ccd2618b734" containerName="istio-proxy" Nov 02 22:02:11.898191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.895716 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3662e24-62ab-48c7-86ae-63a96df54380" containerName="istio-proxy" Nov 02 22:02:11.898332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.898317 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="298aee9e-ab9c-42af-a62c-ef8d48719bff" containerName="istio-proxy" Nov 02 22:02:11.918147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.913785 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:11.946149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.940862 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56k5p\" (UniqueName: \"kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p\") pod \"payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:11.946149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.941187 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:11.950147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:11.949579 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq"] Nov 02 22:02:12.073743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.073560 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:12.233675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.233636 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56k5p\" (UniqueName: \"kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p\") pod \"payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:12.255772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.126753 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-75577e-dockercfg-lxdtt\"" Nov 02 22:02:12.255995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.175815 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:12.256147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.126694 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-75577e\"" Nov 02 22:02:12.259544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.259520 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz"] Nov 02 22:02:12.350196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.334596 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.351098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.351054 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56k5p\" (UniqueName: \"kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p\") pod \"payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:12.354421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.354335 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz"] Nov 02 22:02:12.483636 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.482753 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.483636 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.483166 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlchj\" (UniqueName: \"kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj\") pod \"payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.533198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.533155 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:12.585287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.585250 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.585562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.585544 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlchj\" (UniqueName: \"kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj\") pod \"payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.585952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.585932 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.643334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.643301 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlchj\" (UniqueName: \"kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj\") pod \"payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.731761 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.731725 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:12.923737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:12.923702 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq"] Nov 02 22:02:12.925063 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:12.925031 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0645dc84_b9fb_47d6_b4e2_87772f483148.slice/crio-8a633fb6d91d287320d16a8a8bc3a9448b9dea9c8fe7034a865125810ceec3f5 WatchSource:0}: Error finding container 8a633fb6d91d287320d16a8a8bc3a9448b9dea9c8fe7034a865125810ceec3f5: Status 404 returned error can't find the container with id 8a633fb6d91d287320d16a8a8bc3a9448b9dea9c8fe7034a865125810ceec3f5 Nov 02 22:02:13.117534 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:13.117500 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b403fc0_187f_47a6_83a1_33fc841bca1a.slice/crio-2fd9f7e21d15a52bee1cb4a2095cb5c9cc69c77d1b1dc579c1a8a38b373f4e87 WatchSource:0}: Error finding container 2fd9f7e21d15a52bee1cb4a2095cb5c9cc69c77d1b1dc579c1a8a38b373f4e87: Status 404 returned error can't find the container with id 2fd9f7e21d15a52bee1cb4a2095cb5c9cc69c77d1b1dc579c1a8a38b373f4e87 Nov 02 22:02:13.117713 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.117690 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz"] Nov 02 22:02:13.342204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.342157 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" event={"ID":"3b403fc0-187f-47a6-83a1-33fc841bca1a","Type":"ContainerStarted","Data":"d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0"} Nov 02 22:02:13.342648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.342230 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" event={"ID":"3b403fc0-187f-47a6-83a1-33fc841bca1a","Type":"ContainerStarted","Data":"2fd9f7e21d15a52bee1cb4a2095cb5c9cc69c77d1b1dc579c1a8a38b373f4e87"} Nov 02 22:02:13.342648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.342378 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:13.344156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.344090 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" event={"ID":"0645dc84-b9fb-47d6-b4e2-87772f483148","Type":"ContainerStarted","Data":"03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd"} Nov 02 22:02:13.344156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.344149 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" event={"ID":"0645dc84-b9fb-47d6-b4e2-87772f483148","Type":"ContainerStarted","Data":"8a633fb6d91d287320d16a8a8bc3a9448b9dea9c8fe7034a865125810ceec3f5"} Nov 02 22:02:13.344358 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.344247 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:13.457170 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.457090 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" podStartSLOduration=2.45707691 podStartE2EDuration="2.45707691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:13.45273797 +0000 UTC m=+1672.414288291" watchObservedRunningTime="2026-11-02 22:02:13.45707691 +0000 UTC m=+1672.418627259" Nov 02 22:02:13.629148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:13.623571 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" podStartSLOduration=2.6235447990000003 podStartE2EDuration="2.623544799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:13.580604788 +0000 UTC m=+1672.542155109" watchObservedRunningTime="2026-11-02 22:02:13.623544799 +0000 UTC m=+1672.585095120" Nov 02 22:02:14.985273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:14.985233 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg"] Nov 02 22:02:14.988970 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:14.988938 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:14.995447 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:14.995421 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:14.995559 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:14.995477 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48tz\" (UniqueName: \"kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:14.995629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:14.995607 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:14.995737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:14.995720 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.041707 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.041676 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-fb0367-serving-cert\"" Nov 02 22:02:15.047636 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.047609 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg"] Nov 02 22:02:15.097480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.097443 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.097644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.097540 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48tz\" (UniqueName: \"kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.097644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.097605 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.097772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.097687 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.097884 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:15.097855 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-serving-cert: secret "maas-api-e2e-shared-a-w4-fb0367-serving-cert" not found Nov 02 22:02:15.098001 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:15.097919 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls podName:8a24076f-b642-4d94-bb34-5211faf082ff nodeName:}" failed. No retries permitted until 2026-11-02 22:02:15.597901513 +0000 UTC m=+1674.559451824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls") pod "maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" (UID: "8a24076f-b642-4d94-bb34-5211faf082ff") : secret "maas-api-e2e-shared-a-w4-fb0367-serving-cert" not found Nov 02 22:02:15.098157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.098141 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.100359 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.100337 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.156831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.156763 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48tz\" (UniqueName: \"kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.198831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.198795 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:15.395468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.395387 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:15.557153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.556002 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:15.625592 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.625554 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.628172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.628146 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:15.904406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:15.904375 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:16.018356 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:16.018312 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:02:16.406341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:16.406228 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg"] Nov 02 22:02:16.406482 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:16.406363 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a24076f_b642_4d94_bb34_5211faf082ff.slice/crio-30a086054a29285c6467c0b96ae841cb0b14d2459cc63cdd191af3dbfdab1b69 WatchSource:0}: Error finding container 30a086054a29285c6467c0b96ae841cb0b14d2459cc63cdd191af3dbfdab1b69: Status 404 returned error can't find the container with id 30a086054a29285c6467c0b96ae841cb0b14d2459cc63cdd191af3dbfdab1b69 Nov 02 22:02:16.412017 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:16.411992 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:02:17.009148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:17.008642 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:17.453111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:17.453068 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:17.472055 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:17.472010 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" event={"ID":"8a24076f-b642-4d94-bb34-5211faf082ff","Type":"ContainerStarted","Data":"30a086054a29285c6467c0b96ae841cb0b14d2459cc63cdd191af3dbfdab1b69"} Nov 02 22:02:18.061186 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.035540 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x"] Nov 02 22:02:18.093265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.073413 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.097076 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.129390 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2a9be7-openshift-default-dockercfg-86qgg\"" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130485 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130537 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130591 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130692 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130726 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tc6\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130762 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130872 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.130966 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.159237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.131106 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.170945 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.170912 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x"] Nov 02 22:02:18.265188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.244356 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.275349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.275308 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tc6\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.281261 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.281225 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.291376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.291335 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.359912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.359830 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.360358 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.360328 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.360601 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.360582 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.360741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.360728 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.360877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.360859 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.365983 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.365953 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.366185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.359643 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.366627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.366607 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.366999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.366962 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.367372 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.367352 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.368186 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.368162 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.403075 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.390036 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.411604 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.411534 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:18.449402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.448772 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.452980 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.452924 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tc6\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6\") pod \"e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.532868 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.529773 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:18.709205 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.686209 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" event={"ID":"8a24076f-b642-4d94-bb34-5211faf082ff","Type":"ContainerStarted","Data":"54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a"} Nov 02 22:02:18.709468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:18.709443 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:20.005733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.005673 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" podStartSLOduration=5.009158641 podStartE2EDuration="6.005651024s" totalImagesPullingTime="996.492383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:18.873318 +0000 UTC m=+1677.834868321" watchObservedRunningTime="2026-11-02 22:02:20.005651024 +0000 UTC m=+1678.967201348" Nov 02 22:02:20.010282 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.010176 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x"] Nov 02 22:02:20.080761 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:20.080245 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5e7c02_59cc_4491_8d41_567112f067aa.slice/crio-0b04046d9021b272aa4c29eda55b482c91484bc08360c492461fc921bd2134b0 WatchSource:0}: Error finding container 0b04046d9021b272aa4c29eda55b482c91484bc08360c492461fc921bd2134b0: Status 404 returned error can't find the container with id 0b04046d9021b272aa4c29eda55b482c91484bc08360c492461fc921bd2134b0 Nov 02 22:02:20.083997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.083954 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:20.084217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.084193 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:20.084360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.084341 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:20.891997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.874506 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" event={"ID":"5d5e7c02-59cc-4491-8d41-567112f067aa","Type":"ContainerStarted","Data":"1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b"} Nov 02 22:02:20.891997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:20.874619 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" event={"ID":"5d5e7c02-59cc-4491-8d41-567112f067aa","Type":"ContainerStarted","Data":"0b04046d9021b272aa4c29eda55b482c91484bc08360c492461fc921bd2134b0"} Nov 02 22:02:21.146567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:21.146457 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podStartSLOduration=4.146436349 podStartE2EDuration="4.146436349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:21.137345103 +0000 UTC m=+1680.098895425" watchObservedRunningTime="2026-11-02 22:02:21.146436349 +0000 UTC m=+1680.107986670" Nov 02 22:02:21.387519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:21.387488 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:21.535516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:21.535467 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:21.560525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:21.560491 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:21.882432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:21.882345 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:21.883701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:21.883680 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:02:22.367833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:22.367799 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:22.738564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:22.738531 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:22.818908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:22.818876 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk"] Nov 02 22:02:22.849644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:22.849609 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:22.908446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:22.908415 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4fc235-dockercfg-n4q47\"" Nov 02 22:02:22.914158 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:22.909331 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4fc235\"" Nov 02 22:02:23.151907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.151817 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnw4v\" (UniqueName: \"kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v\") pod \"payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.153274 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.153243 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.270145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.153915 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk"] Nov 02 22:02:23.343467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.343424 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnw4v\" (UniqueName: \"kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v\") pod \"payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.343868 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.343848 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.345300 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.345275 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.422411 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.421610 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnw4v\" (UniqueName: \"kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v\") pod \"payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.563954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.563920 4285 scope.go:122] "RemoveContainer" containerID="5f217076373359ed3ac6a38f5d389cb1bc1e11e13168ae861e0a3fea136d2fd1" Nov 02 22:02:23.597883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.597851 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:23.848064 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.848011 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:23.989985 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:23.989949 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad35d428_e5de_4823_a526_fb35b9f63455.slice/crio-5a07c73f22a67659e4c4591ff1f1f29f1c1bc1dba27ca38db2ae3f91cb5f739c WatchSource:0}: Error finding container 5a07c73f22a67659e4c4591ff1f1f29f1c1bc1dba27ca38db2ae3f91cb5f739c: Status 404 returned error can't find the container with id 5a07c73f22a67659e4c4591ff1f1f29f1c1bc1dba27ca38db2ae3f91cb5f739c Nov 02 22:02:23.990099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:23.989998 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk"] Nov 02 22:02:24.004581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.004556 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:24.166026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.165951 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:24.353655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.353626 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:02:24.353834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.353694 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:02:24.428675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.428581 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" event={"ID":"ad35d428-e5de-4823-a526-fb35b9f63455","Type":"ContainerStarted","Data":"0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac"} Nov 02 22:02:24.428675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.428642 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:24.428675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.428660 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" event={"ID":"ad35d428-e5de-4823-a526-fb35b9f63455","Type":"ContainerStarted","Data":"5a07c73f22a67659e4c4591ff1f1f29f1c1bc1dba27ca38db2ae3f91cb5f739c"} Nov 02 22:02:24.704201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.703725 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" podStartSLOduration=2.703704078 podStartE2EDuration="2.703704078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:24.701788528 +0000 UTC m=+1683.663338848" watchObservedRunningTime="2026-11-02 22:02:24.703704078 +0000 UTC m=+1683.665254399" Nov 02 22:02:24.801456 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:24.800311 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:02:26.552854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:26.552823 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:31.481418 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:31.481378 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:31.912477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:31.912397 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:32.267151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.241477 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:32.497149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.490990 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz"] Nov 02 22:02:32.508747 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.508719 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:32.624148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.603544 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz"] Nov 02 22:02:32.624703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.605093 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:32.626200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.626166 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmpx\" (UniqueName: \"kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx\") pod \"payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:32.626900 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.612118 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2a9be7-dockercfg-99lpn\"" Nov 02 22:02:32.627035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.611642 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2a9be7\"" Nov 02 22:02:32.732774 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.732739 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmpx\" (UniqueName: \"kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx\") pod \"payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:32.775830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.775773 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:32.779412 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.778357 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:32.827340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:32.827307 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmpx\" (UniqueName: \"kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx\") pod \"payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:33.032544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:33.031453 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:33.455390 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:33.455358 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e06ba03_0bfc_48c2_be32_be8c4f0cdd0b.slice/crio-7c6fd2024a4eeef3e2ec62f3e1684ae52c763bfc7df8c9e935b1f46c4779bd7a WatchSource:0}: Error finding container 7c6fd2024a4eeef3e2ec62f3e1684ae52c763bfc7df8c9e935b1f46c4779bd7a: Status 404 returned error can't find the container with id 7c6fd2024a4eeef3e2ec62f3e1684ae52c763bfc7df8c9e935b1f46c4779bd7a Nov 02 22:02:33.455508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:33.455456 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz"] Nov 02 22:02:33.791691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:33.791606 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" event={"ID":"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b","Type":"ContainerStarted","Data":"17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038"} Nov 02 22:02:33.791691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:33.791663 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" event={"ID":"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b","Type":"ContainerStarted","Data":"7c6fd2024a4eeef3e2ec62f3e1684ae52c763bfc7df8c9e935b1f46c4779bd7a"} Nov 02 22:02:33.791691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:33.791690 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:33.907565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:33.907505 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" podStartSLOduration=1.907491338 podStartE2EDuration="1.907491338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:33.903357641 +0000 UTC m=+1692.864907980" watchObservedRunningTime="2026-11-02 22:02:33.907491338 +0000 UTC m=+1692.869041672" Nov 02 22:02:35.439434 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:35.439401 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:02:36.256454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.256418 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb"] Nov 02 22:02:36.261427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.261406 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.316993 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.316959 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb"] Nov 02 22:02:36.360997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.360960 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-204000-serving-cert\"" Nov 02 22:02:36.380183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.380146 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.380377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.380355 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.380520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.380497 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.380630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.380614 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghl2n\" (UniqueName: \"kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.481642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.481605 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.482112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.481701 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.482112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.481774 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghl2n\" (UniqueName: \"kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.482112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.481828 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.482112 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:36.481978 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-serving-cert: secret "maas-api-e2e-shared-a-w2-204000-serving-cert" not found Nov 02 22:02:36.482112 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:36.482071 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls podName:3278a470-3837-4abd-baf6-de3a96e2371d nodeName:}" failed. No retries permitted until 2026-11-02 22:02:36.982047359 +0000 UTC m=+1695.943597662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" (UID: "3278a470-3837-4abd-baf6-de3a96e2371d") : secret "maas-api-e2e-shared-a-w2-204000-serving-cert" not found Nov 02 22:02:36.482464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.482441 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.484381 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.484364 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.489782 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.489759 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:36.581310 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.575073 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghl2n\" (UniqueName: \"kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.652262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.652223 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:36.806517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.806481 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:36.995971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.995939 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:36.998360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:36.998336 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:37.176654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:37.176627 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:37.556600 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:02:37.556554 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3278a470_3837_4abd_baf6_de3a96e2371d.slice/crio-43b396870e21d6e8fd806bd83d54a19133a83cf6598d2f2e12d1d06e6cf28c65 WatchSource:0}: Error finding container 43b396870e21d6e8fd806bd83d54a19133a83cf6598d2f2e12d1d06e6cf28c65: Status 404 returned error can't find the container with id 43b396870e21d6e8fd806bd83d54a19133a83cf6598d2f2e12d1d06e6cf28c65 Nov 02 22:02:37.557025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:37.557003 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb"] Nov 02 22:02:37.894715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:37.894635 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" event={"ID":"3278a470-3837-4abd-baf6-de3a96e2371d","Type":"ContainerStarted","Data":"43b396870e21d6e8fd806bd83d54a19133a83cf6598d2f2e12d1d06e6cf28c65"} Nov 02 22:02:38.904187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:38.904144 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" event={"ID":"3278a470-3837-4abd-baf6-de3a96e2371d","Type":"ContainerStarted","Data":"53d0d4b2f68c91e84f03c7d76bf195da565826b92bf7326d7227232280b14d28"} Nov 02 22:02:38.904769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:38.904210 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:39.018058 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:39.017998 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" podStartSLOduration=2.214829057 podStartE2EDuration="3.01798239s" totalImagesPullingTime="803.153333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:39.016205462 +0000 UTC m=+1697.977755776" watchObservedRunningTime="2026-11-02 22:02:39.01798239 +0000 UTC m=+1697.979532757" Nov 02 22:02:41.860546 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:41.860512 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:44.857849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:44.857820 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:02:44.917528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:44.917499 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:02:45.500813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.500784 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb"] Nov 02 22:02:45.501003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.500973 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" containerID="cri-o://5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f" gracePeriod=30 Nov 02 22:02:45.505474 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.505447 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6"] Nov 02 22:02:45.505645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.505626 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" containerID="cri-o://c3ed6149a880a0e5d42ba592dafe35dfd9610567ad14fc11b8539cccae39001c" gracePeriod=30 Nov 02 22:02:45.556771 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.556731 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz"] Nov 02 22:02:45.556988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.556961 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" containerID="cri-o://bacf9fd4d78c97717bd7691cf55086cea24b8d9f72ad13d12c4eaf5dfd9184f1" gracePeriod=30 Nov 02 22:02:45.559494 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.559465 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9"] Nov 02 22:02:45.559660 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.559639 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" containerID="cri-o://0d4597c80f5a9090fdb5fe8a966248baaff5e06bf342d8a313b478d7247608f3" gracePeriod=30 Nov 02 22:02:45.669937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.669902 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf"] Nov 02 22:02:45.670279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.670229 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" containerID="cri-o://b1c593dfb9f984335cb37303c886a788cd6b4a745ce926eef2e22a82be952265" gracePeriod=30 Nov 02 22:02:45.675229 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.675182 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj"] Nov 02 22:02:45.675536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.675481 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" containerID="cri-o://e5330c2c0bc5386983a022108b2a33dc7397474c2e3445e6dd68418a2a808a3b" gracePeriod=30 Nov 02 22:02:45.681428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.680950 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2"] Nov 02 22:02:45.681428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.681197 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" containerID="cri-o://7f4b6a58218d0512f5af4a0e0bc84d6136dd7e0ff4d18ebfb0b15ca3a3106502" gracePeriod=30 Nov 02 22:02:45.685984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.685956 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2"] Nov 02 22:02:45.686233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:45.686160 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" containerID="cri-o://741e4bf961bd10f032e0c9e377d835204d91f7e5395dc092c5f5d5253ce927df" gracePeriod=30 Nov 02 22:02:46.011335 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:46.011298 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:02:46.111009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:46.110979 4285 patch_prober.go:29] interesting pod/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:02:46.111188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:46.111031 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:02:47.032964 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:47.032923 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:51.021585 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.017227 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.082852 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.082952 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083015 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083092 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083163 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083296 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083441 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083600 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.083739 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token\") pod \"acb34325-144e-449c-be36-7240ee0f4acd\" (UID: \"acb34325-144e-449c-be36-7240ee0f4acd\") " Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.084504 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.084744 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085018 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085066 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085209 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085766 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085790 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085820 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb34325-144e-449c-be36-7240ee0f4acd-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085834 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.086414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.085847 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.089854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.089794 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "kube-api-access-thzjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.090220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.090159 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.090312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.090260 4285 generic.go:356] "Generic (PLEG): container finished" podID="9e29ece0-7361-4bae-8651-18356ed45965" containerID="e5330c2c0bc5386983a022108b2a33dc7397474c2e3445e6dd68418a2a808a3b" exitCode=0 Nov 02 22:02:51.090312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.090298 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.090436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.090350 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" event={"ID":"9e29ece0-7361-4bae-8651-18356ed45965","Type":"ContainerDied","Data":"e5330c2c0bc5386983a022108b2a33dc7397474c2e3445e6dd68418a2a808a3b"} Nov 02 22:02:51.090986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.090935 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy" pod "acb34325-144e-449c-be36-7240ee0f4acd" (UID: "acb34325-144e-449c-be36-7240ee0f4acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.091572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.091556 4285 scope.go:122] "RemoveContainer" containerID="2740ef9f1a718fc9c17d7684fcfedc15c552b8c307d5bc9510bc4008eb18f940" Nov 02 22:02:51.093164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.092943 4285 generic.go:356] "Generic (PLEG): container finished" podID="0e6c42fc-46b8-4c30-9129-691f98079984" containerID="741e4bf961bd10f032e0c9e377d835204d91f7e5395dc092c5f5d5253ce927df" exitCode=0 Nov 02 22:02:51.093164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.093026 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" event={"ID":"0e6c42fc-46b8-4c30-9129-691f98079984","Type":"ContainerDied","Data":"741e4bf961bd10f032e0c9e377d835204d91f7e5395dc092c5f5d5253ce927df"} Nov 02 22:02:51.100628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.099535 4285 generic.go:356] "Generic (PLEG): container finished" podID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerID="7f4b6a58218d0512f5af4a0e0bc84d6136dd7e0ff4d18ebfb0b15ca3a3106502" exitCode=0 Nov 02 22:02:51.100628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.099695 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" event={"ID":"e3f0ca1c-72dc-4d22-96e2-547800ce142f","Type":"ContainerDied","Data":"7f4b6a58218d0512f5af4a0e0bc84d6136dd7e0ff4d18ebfb0b15ca3a3106502"} Nov 02 22:02:51.103978 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.103958 4285 generic.go:356] "Generic (PLEG): container finished" podID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerID="c3ed6149a880a0e5d42ba592dafe35dfd9610567ad14fc11b8539cccae39001c" exitCode=0 Nov 02 22:02:51.104101 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.104045 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" event={"ID":"a1add337-86c2-4e46-b123-b29ed4f3895f","Type":"ContainerDied","Data":"c3ed6149a880a0e5d42ba592dafe35dfd9610567ad14fc11b8539cccae39001c"} Nov 02 22:02:51.106006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.105989 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 22:02:51.106843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.106819 4285 generic.go:356] "Generic (PLEG): container finished" podID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerID="bacf9fd4d78c97717bd7691cf55086cea24b8d9f72ad13d12c4eaf5dfd9184f1" exitCode=0 Nov 02 22:02:51.106976 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.106883 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" event={"ID":"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f","Type":"ContainerDied","Data":"bacf9fd4d78c97717bd7691cf55086cea24b8d9f72ad13d12c4eaf5dfd9184f1"} Nov 02 22:02:51.119403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.110260 4285 generic.go:356] "Generic (PLEG): container finished" podID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerID="0d4597c80f5a9090fdb5fe8a966248baaff5e06bf342d8a313b478d7247608f3" exitCode=0 Nov 02 22:02:51.119403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.110432 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" event={"ID":"303e2c16-f572-4d46-bb24-57ca0d3b83fa","Type":"ContainerDied","Data":"0d4597c80f5a9090fdb5fe8a966248baaff5e06bf342d8a313b478d7247608f3"} Nov 02 22:02:51.119403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.114575 4285 generic.go:356] "Generic (PLEG): container finished" podID="acb34325-144e-449c-be36-7240ee0f4acd" containerID="5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f" exitCode=0 Nov 02 22:02:51.119403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.114653 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" event={"ID":"acb34325-144e-449c-be36-7240ee0f4acd","Type":"ContainerDied","Data":"5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f"} Nov 02 22:02:51.119403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.114699 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" event={"ID":"acb34325-144e-449c-be36-7240ee0f4acd","Type":"ContainerDied","Data":"525a84f5fa62cbc818e61c3d72b2ffa33400243df793f8d80c4f9d8bbef917b6"} Nov 02 22:02:51.119403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.114778 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb" Nov 02 22:02:51.122371 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.122290 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 22:02:51.127331 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.127114 4285 generic.go:356] "Generic (PLEG): container finished" podID="20b9e799-217d-4d17-b996-a387a1e17101" containerID="b1c593dfb9f984335cb37303c886a788cd6b4a745ce926eef2e22a82be952265" exitCode=0 Nov 02 22:02:51.127331 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.127197 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" event={"ID":"20b9e799-217d-4d17-b996-a387a1e17101","Type":"ContainerDied","Data":"b1c593dfb9f984335cb37303c886a788cd6b4a745ce926eef2e22a82be952265"} Nov 02 22:02:51.130441 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.130410 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 22:02:51.131047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.130791 4285 scope.go:122] "RemoveContainer" containerID="5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f" Nov 02 22:02:51.177393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.176299 4285 scope.go:122] "RemoveContainer" containerID="82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54" Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187419 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187456 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187486 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187520 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187605 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187635 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187691 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.187963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.187717 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.193207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.192225 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.193207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.192954 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.193471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193434 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193627 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193661 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193690 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193707 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193757 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193791 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193821 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193845 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193870 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193932 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.193978 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.194033 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.194233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.194076 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.195494 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.195211 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.196200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.195927 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.197226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.195703 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.197226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.196397 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.197226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.196439 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.197226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.196467 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert\") pod \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\" (UID: \"303e2c16-f572-4d46-bb24-57ca0d3b83fa\") " Nov 02 22:02:51.197226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.196515 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket\") pod \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\" (UID: \"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f\") " Nov 02 22:02:51.197226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.196551 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data\") pod \"a1add337-86c2-4e46-b123-b29ed4f3895f\" (UID: \"a1add337-86c2-4e46-b123-b29ed4f3895f\") " Nov 02 22:02:51.198045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.197807 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.198527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.198504 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzjm\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-kube-api-access-thzjm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.198661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.198644 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.198765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.198752 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.198854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.198842 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.198939 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.198927 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb34325-144e-449c-be36-7240ee0f4acd-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.199021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.199009 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb34325-144e-449c-be36-7240ee0f4acd-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.199103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.199091 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.199282 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.199263 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.199363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.199297 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb34325-144e-449c-be36-7240ee0f4acd-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.199363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.199313 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.201617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.201483 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "kube-api-access-nssdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.201849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.201725 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.201948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.201935 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.202269 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.202249 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.202504 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.202490 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.202791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.202778 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.205934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.205492 4285 scope.go:122] "RemoveContainer" containerID="5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f" Nov 02 22:02:51.206137 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:51.206101 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f\": container with ID starting with 5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f not found: ID does not exist" containerID="5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f" Nov 02 22:02:51.206231 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.206150 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f"} err="failed to get container status \"5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f\": rpc error: code = NotFound desc = could not find container \"5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f\": container with ID starting with 5ada08514727b2e809187ac3f1e2192e556449b17985a760ca849faa51d0096f not found: ID does not exist" Nov 02 22:02:51.206231 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.206179 4285 scope.go:122] "RemoveContainer" containerID="82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54" Nov 02 22:02:51.206359 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.206284 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.206722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.206695 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.207834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.206951 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.207834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.206769 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.207834 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:02:51.207312 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54\": container with ID starting with 82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54 not found: ID does not exist" containerID="82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54" Nov 02 22:02:51.207834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.207349 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54"} err="failed to get container status \"82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54\": rpc error: code = NotFound desc = could not find container \"82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54\": container with ID starting with 82d54b22771639c26654d8da39b79449cfaf6d0b68a3310640402ccf52f4ba54 not found: ID does not exist" Nov 02 22:02:51.210401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.210324 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.210401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.210330 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.210705 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.210667 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.212203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.211646 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.212203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.211960 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.214182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.212784 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "kube-api-access-m9p6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.214182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.212868 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.214182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.212953 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "kube-api-access-xhppj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.214182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.213026 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token" pod "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" (UID: "c032dac6-27b6-4f27-8a22-2d3df7ca0f6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.214591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.214558 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token" pod "a1add337-86c2-4e46-b123-b29ed4f3895f" (UID: "a1add337-86c2-4e46-b123-b29ed4f3895f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.277846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.276454 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token" pod "303e2c16-f572-4d46-bb24-57ca0d3b83fa" (UID: "303e2c16-f572-4d46-bb24-57ca0d3b83fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.293583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.293550 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 22:02:51.300343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.300241 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301856 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p6p\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-kube-api-access-m9p6p\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301883 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhppj\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-kube-api-access-xhppj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301900 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301913 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301927 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301941 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301955 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301970 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301984 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.301999 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302014 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302029 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302044 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add337-86c2-4e46-b123-b29ed4f3895f-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302061 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302073 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302087 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302098 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302107 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add337-86c2-4e46-b123-b29ed4f3895f-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302115 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nssdb\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-kube-api-access-nssdb\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302141 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.302196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.302154 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/303e2c16-f572-4d46-bb24-57ca0d3b83fa-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.305862 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.305806 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 22:02:51.387063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.387034 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 22:02:51.403299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403274 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.403457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403342 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.403457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403376 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.403457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403412 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.403644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403461 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.403644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403487 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.403644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403538 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.403644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403576 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.403644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403604 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.403644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403637 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.403942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403659 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.403942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403665 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.403942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403774 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.403942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.403877 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.404163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404001 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.404163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404042 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.404163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404082 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.404163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404144 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.404163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404118 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.404425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404174 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.404425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404201 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.404425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404316 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.404425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404350 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.404425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404407 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.404686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404441 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.404686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404517 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo\") pod \"0e6c42fc-46b8-4c30-9129-691f98079984\" (UID: \"0e6c42fc-46b8-4c30-9129-691f98079984\") " Nov 02 22:02:51.404686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404555 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.404686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404603 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.404686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404315 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.404686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404492 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.404874 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404599 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.404874 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404848 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.405084 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.405063 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.405198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.405153 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.405264 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.405212 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.405913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.405452 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.409384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.409360 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.428529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.427407 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.435217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.435183 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.435420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.435397 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.404635 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy\") pod \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\" (UID: \"e3f0ca1c-72dc-4d22-96e2-547800ce142f\") " Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.436070 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo\") pod \"20b9e799-217d-4d17-b996-a387a1e17101\" (UID: \"20b9e799-217d-4d17-b996-a387a1e17101\") " Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437032 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437056 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437070 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437084 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437098 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.437189 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437114 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437757 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437779 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437804 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437818 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437833 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437847 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437861 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c42fc-46b8-4c30-9129-691f98079984-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437876 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.438188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.437891 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b9e799-217d-4d17-b996-a387a1e17101-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.443708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.439220 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.443708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.440771 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.443708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.443677 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.443934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.443912 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.444600 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.444577 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.444719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.444644 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.446464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.446425 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.446464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.446425 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.446612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.446591 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl" pod "20b9e799-217d-4d17-b996-a387a1e17101" (UID: "20b9e799-217d-4d17-b996-a387a1e17101"). InnerVolumeSpecName "kube-api-access-kb4hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.447611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.447580 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "kube-api-access-bqgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.449347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.449311 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k" pod "0e6c42fc-46b8-4c30-9129-691f98079984" (UID: "0e6c42fc-46b8-4c30-9129-691f98079984"). InnerVolumeSpecName "kube-api-access-4cd5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.449563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.449539 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token" pod "e3f0ca1c-72dc-4d22-96e2-547800ce142f" (UID: "e3f0ca1c-72dc-4d22-96e2-547800ce142f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.539541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.539507 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.540006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.539979 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.546041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546004 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.546168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546085 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.546234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546202 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.546491 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.542386 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:51.546879 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546702 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.546879 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546693 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.546879 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546817 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.547287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546885 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.547287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.546954 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.547287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.547085 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs\") pod \"9e29ece0-7361-4bae-8651-18356ed45965\" (UID: \"9e29ece0-7361-4bae-8651-18356ed45965\") " Nov 02 22:02:51.547479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.547324 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:51.547679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.547545 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.547755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.547717 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.548000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.547977 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548007 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548023 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548037 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548051 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6c42fc-46b8-4c30-9129-691f98079984-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548065 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4hl\" (UniqueName: \"kubernetes.io/projected/20b9e799-217d-4d17-b996-a387a1e17101-kube-api-access-kb4hl\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548079 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e29ece0-7361-4bae-8651-18356ed45965-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548097 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548111 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b9e799-217d-4d17-b996-a387a1e17101-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548139 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548155 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6c42fc-46b8-4c30-9129-691f98079984-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548170 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgpb\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-kube-api-access-bqgpb\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548184 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cd5k\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-kube-api-access-4cd5k\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548198 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548212 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6c42fc-46b8-4c30-9129-691f98079984-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548225 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b9e799-217d-4d17-b996-a387a1e17101-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548238 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e29ece0-7361-4bae-8651-18356ed45965-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.548423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.548251 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f0ca1c-72dc-4d22-96e2-547800ce142f-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.549413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.549376 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:51.549937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.549909 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "kube-api-access-nz9jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.552623 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.552595 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token" pod "9e29ece0-7361-4bae-8651-18356ed45965" (UID: "9e29ece0-7361-4bae-8651-18356ed45965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:51.559094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.559041 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb"] Nov 02 22:02:51.564495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.564471 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-bg6gb"] Nov 02 22:02:51.650804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.650721 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz9jf\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-kube-api-access-nz9jf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.650804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.650762 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e29ece0-7361-4bae-8651-18356ed45965-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:51.650804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:51.650783 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e29ece0-7361-4bae-8651-18356ed45965-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:02:52.098797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.098740 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:52.135814 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.135786 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" Nov 02 22:02:52.135987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.135789 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj" event={"ID":"9e29ece0-7361-4bae-8651-18356ed45965","Type":"ContainerDied","Data":"679bda35659e4b7b9db77d03f9942817e458c2084fb5b518e06434f1434d3ad0"} Nov 02 22:02:52.136047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.135990 4285 scope.go:122] "RemoveContainer" containerID="e5330c2c0bc5386983a022108b2a33dc7397474c2e3445e6dd68418a2a808a3b" Nov 02 22:02:52.137455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.137435 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" Nov 02 22:02:52.137551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.137447 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2" event={"ID":"0e6c42fc-46b8-4c30-9129-691f98079984","Type":"ContainerDied","Data":"b264f6f7728d50f296907906718901cd23d4303e205186f5d111e55b2d7780e2"} Nov 02 22:02:52.139261 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.139229 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" event={"ID":"e3f0ca1c-72dc-4d22-96e2-547800ce142f","Type":"ContainerDied","Data":"8037e2053409c498fda8c4ef11899dc7561b50b6f391690799a95179bb8fd0de"} Nov 02 22:02:52.139408 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.139251 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2" Nov 02 22:02:52.140745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.140717 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" event={"ID":"a1add337-86c2-4e46-b123-b29ed4f3895f","Type":"ContainerDied","Data":"f43f67cd024443771fcee71648fa2000227ac32d6a2469be3ae60b9c0d04ae70"} Nov 02 22:02:52.140859 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.140822 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6" Nov 02 22:02:52.147995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.147750 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" Nov 02 22:02:52.147995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.147768 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz" event={"ID":"c032dac6-27b6-4f27-8a22-2d3df7ca0f6f","Type":"ContainerDied","Data":"6dfd3ac8314342de37daaa82fd4d94886972174ca7a0d88e194a0c9fc2fa2190"} Nov 02 22:02:52.149609 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.149591 4285 scope.go:122] "RemoveContainer" containerID="741e4bf961bd10f032e0c9e377d835204d91f7e5395dc092c5f5d5253ce927df" Nov 02 22:02:52.150448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.150423 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" event={"ID":"303e2c16-f572-4d46-bb24-57ca0d3b83fa","Type":"ContainerDied","Data":"00596ffaffa642817159101ece57c452ccfe4871362e9e4822864a4befc826a2"} Nov 02 22:02:52.150448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.150438 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9" Nov 02 22:02:52.155743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.155712 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" event={"ID":"20b9e799-217d-4d17-b996-a387a1e17101","Type":"ContainerDied","Data":"d070f678e176675bf09b3b157cf198fc32481ec871c18e657c8665f518a336b2"} Nov 02 22:02:52.155822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.155749 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf" Nov 02 22:02:52.163894 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.163806 4285 scope.go:122] "RemoveContainer" containerID="7f4b6a58218d0512f5af4a0e0bc84d6136dd7e0ff4d18ebfb0b15ca3a3106502" Nov 02 22:02:52.260602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.259712 4285 scope.go:122] "RemoveContainer" containerID="c3ed6149a880a0e5d42ba592dafe35dfd9610567ad14fc11b8539cccae39001c" Nov 02 22:02:52.305374 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.302785 4285 scope.go:122] "RemoveContainer" containerID="bacf9fd4d78c97717bd7691cf55086cea24b8d9f72ad13d12c4eaf5dfd9184f1" Nov 02 22:02:52.327911 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.327450 4285 scope.go:122] "RemoveContainer" containerID="0d4597c80f5a9090fdb5fe8a966248baaff5e06bf342d8a313b478d7247608f3" Nov 02 22:02:52.338350 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.338321 4285 scope.go:122] "RemoveContainer" containerID="b1c593dfb9f984335cb37303c886a788cd6b4a745ce926eef2e22a82be952265" Nov 02 22:02:52.529477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.529439 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2"] Nov 02 22:02:52.533539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.533511 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-vt7d2"] Nov 02 22:02:52.640040 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.640008 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6"] Nov 02 22:02:52.643910 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.643883 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-l2fb6"] Nov 02 22:02:52.752250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.752218 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj"] Nov 02 22:02:52.755733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.755712 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-b27nj"] Nov 02 22:02:52.871037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.870855 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9"] Nov 02 22:02:52.879094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.879071 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-rn6j9"] Nov 02 22:02:52.990235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.990202 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2"] Nov 02 22:02:52.993649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:52.993624 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-n6qz2"] Nov 02 22:02:53.103247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.103216 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz"] Nov 02 22:02:53.109428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.109399 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-64438f-openshift-default-6996f8785-dj5jz"] Nov 02 22:02:53.175626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.175563 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" path="/var/lib/kubelet/pods/0e6c42fc-46b8-4c30-9129-691f98079984/volumes" Nov 02 22:02:53.176050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.176034 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" path="/var/lib/kubelet/pods/303e2c16-f572-4d46-bb24-57ca0d3b83fa/volumes" Nov 02 22:02:53.176497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.176482 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e29ece0-7361-4bae-8651-18356ed45965" path="/var/lib/kubelet/pods/9e29ece0-7361-4bae-8651-18356ed45965/volumes" Nov 02 22:02:53.177261 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.177246 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" path="/var/lib/kubelet/pods/a1add337-86c2-4e46-b123-b29ed4f3895f/volumes" Nov 02 22:02:53.177690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.177676 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb34325-144e-449c-be36-7240ee0f4acd" path="/var/lib/kubelet/pods/acb34325-144e-449c-be36-7240ee0f4acd/volumes" Nov 02 22:02:53.178133 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.178111 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" path="/var/lib/kubelet/pods/c032dac6-27b6-4f27-8a22-2d3df7ca0f6f/volumes" Nov 02 22:02:53.178842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.178827 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" path="/var/lib/kubelet/pods/e3f0ca1c-72dc-4d22-96e2-547800ce142f/volumes" Nov 02 22:02:53.215693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.215663 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf"] Nov 02 22:02:53.221866 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:53.221836 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ffe386-openshift-default-756448cc4c-69pdf"] Nov 02 22:02:55.179293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:55.179259 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b9e799-217d-4d17-b996-a387a1e17101" path="/var/lib/kubelet/pods/20b9e799-217d-4d17-b996-a387a1e17101/volumes" Nov 02 22:02:56.798914 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:02:56.798876 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:01.434574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:01.434541 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:05.766625 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:05.766590 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:06.202420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:06.202392 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:10.903266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:10.903216 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:11.607363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:11.607329 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:11.915620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:11.915531 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:15.549222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:15.549169 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:15.709426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:15.709397 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:15.879857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:15.879779 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:16.007791 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:03:16.007759 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:03:20.303834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:20.303786 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:20.458883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:20.458852 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:20.763503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:20.763472 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:25.149098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:25.149060 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:29.924798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:29.924744 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:34.697762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:34.697672 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:37.733640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:37.733602 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:38.147719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:38.147630 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:38.860148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:38.841895 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:38.863673 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:38.863636 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:03:38.959787 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:38.959668 4285 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 02 22:03:39.029160 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:39.023805 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 02 22:03:39.124637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:39.112462 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:40.146707 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.146676 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz"] Nov 02 22:03:40.147590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147562 4285 cpu_manager.go:414] "containerMap: removing container" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" Nov 02 22:03:40.147590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147590 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147609 4285 cpu_manager.go:414] "containerMap: removing container" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147616 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147639 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147646 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147670 4285 cpu_manager.go:414] "containerMap: removing container" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147676 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147708 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147716 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147733 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147740 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147753 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147760 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147772 4285 cpu_manager.go:414] "containerMap: removing container" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" Nov 02 22:03:40.147768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147778 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147803 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147810 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147823 4285 cpu_manager.go:414] "containerMap: removing container" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.147830 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148009 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="c032dac6-27b6-4f27-8a22-2d3df7ca0f6f" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148024 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148039 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6c42fc-46b8-4c30-9129-691f98079984" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148050 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b9e799-217d-4d17-b996-a387a1e17101" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148066 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148078 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e29ece0-7361-4bae-8651-18356ed45965" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148088 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f0ca1c-72dc-4d22-96e2-547800ce142f" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148097 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="303e2c16-f572-4d46-bb24-57ca0d3b83fa" containerName="istio-proxy" Nov 02 22:03:40.148431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.148108 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1add337-86c2-4e46-b123-b29ed4f3895f" containerName="istio-proxy" Nov 02 22:03:40.152532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.152512 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.205747 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.205720 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz"] Nov 02 22:03:40.250287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.250259 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2a9be7-serving-cert\"" Nov 02 22:03:40.257308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.255051 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.257308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.255119 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.257308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.255283 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jct\" (UniqueName: \"kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.257308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.255416 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.298004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.297976 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:40.356784 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.356751 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.356969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.356802 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.356969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.356838 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jct\" (UniqueName: \"kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.357065 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.356995 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.357643 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.357612 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.359470 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.359443 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.359615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.359577 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.450262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.450229 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jct\" (UniqueName: \"kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct\") pod \"maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:40.469739 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:40.469713 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:41.080756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:41.080725 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz"] Nov 02 22:03:41.081457 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:03:41.081425 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbd8b2b_c40f_462a_8f5d_31c92e17f210.slice/crio-ea0c7063b6c547c0c9fbb1f76e6764b0d67a15e377e6af4641f0af2b66c0692a WatchSource:0}: Error finding container ea0c7063b6c547c0c9fbb1f76e6764b0d67a15e377e6af4641f0af2b66c0692a: Status 404 returned error can't find the container with id ea0c7063b6c547c0c9fbb1f76e6764b0d67a15e377e6af4641f0af2b66c0692a Nov 02 22:03:41.650221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:41.650185 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" event={"ID":"bfbd8b2b-c40f-462a-8f5d-31c92e17f210","Type":"ContainerStarted","Data":"ea0c7063b6c547c0c9fbb1f76e6764b0d67a15e377e6af4641f0af2b66c0692a"} Nov 02 22:03:42.378568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:42.378536 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:42.659767 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:42.659672 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" event={"ID":"bfbd8b2b-c40f-462a-8f5d-31c92e17f210","Type":"ContainerStarted","Data":"9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11"} Nov 02 22:03:42.659767 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:42.659736 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:42.775839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:42.775791 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" podStartSLOduration=1.973334195 podStartE2EDuration="2.77577723s" totalImagesPullingTime="802.443035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:42.771745491 +0000 UTC m=+1761.733295809" watchObservedRunningTime="2026-11-02 22:03:42.77577723 +0000 UTC m=+1761.737327549" Nov 02 22:03:43.608315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:43.608281 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:43.850621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:43.850590 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:45.577656 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:45.577578 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:46.007833 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:03:46.007791 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:03:47.714259 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:47.714223 4285 generic.go:356] "Generic (PLEG): container finished" podID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerID="4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec" exitCode=0 Nov 02 22:03:47.714724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:47.714306 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" event={"ID":"6dbf506d-f704-4229-9d75-e2be5bd77925","Type":"ContainerDied","Data":"4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec"} Nov 02 22:03:47.714805 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:47.714721 4285 scope.go:122] "RemoveContainer" containerID="4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec" Nov 02 22:03:47.716539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:47.716508 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:47.716640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:47.716595 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:47.716682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:47.716640 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:48.573590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:48.573557 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:48.761667 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:48.719334 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:03:48.931249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:48.877952 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:49.588339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.588307 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" event={"ID":"6dbf506d-f704-4229-9d75-e2be5bd77925","Type":"ContainerStarted","Data":"17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b"} Nov 02 22:03:49.588570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.588555 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb"] Nov 02 22:03:49.589532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.589511 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb34325-144e-449c-be36-7240ee0f4acd" containerName="istio-proxy" Nov 02 22:03:49.592140 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.592111 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb"] Nov 02 22:03:49.592226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.592191 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.652557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.652478 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-267d89\"" Nov 02 22:03:49.689648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.689603 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-267d89-dockercfg-pjtlv\"" Nov 02 22:03:49.709490 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.709460 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczkm\" (UniqueName: \"kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm\") pod \"payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.709634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.709533 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.811537 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.811504 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczkm\" (UniqueName: \"kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm\") pod \"payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.811988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.811591 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.825153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.812583 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.867964 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.867890 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczkm\" (UniqueName: \"kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm\") pod \"payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:49.906529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:49.906500 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:50.339925 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.339892 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb"] Nov 02 22:03:50.545333 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.545258 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:03:50.547006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.545632 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 22:03:50.547006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.545696 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 22:03:50.603064 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.602979 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" event={"ID":"b1c42c48-894a-478b-858e-304f21ebcbff","Type":"ContainerStarted","Data":"c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666"} Nov 02 22:03:50.603064 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.603043 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" event={"ID":"b1c42c48-894a-478b-858e-304f21ebcbff","Type":"ContainerStarted","Data":"ba01a55b3665a4201d4e363c3aa487585ca195dabd9c1eb7381f9a79b97b659f"} Nov 02 22:03:50.603266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.603093 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:03:50.702120 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.702041 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:50.731950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:50.731890 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" podStartSLOduration=1.731872259 podStartE2EDuration="1.731872259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:50.727494654 +0000 UTC m=+1769.689044973" watchObservedRunningTime="2026-11-02 22:03:50.731872259 +0000 UTC m=+1769.693422578" Nov 02 22:03:51.275705 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:51.275672 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:03:51.541687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:51.541599 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 22:03:51.541946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:51.541920 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 22:03:52.543282 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:52.543237 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 22:03:52.543816 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:52.543298 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 22:03:53.547541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:53.547508 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 22:03:53.548028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:53.547994 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 22:03:54.545919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:54.545885 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:03:54.546781 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:54.546756 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:03:55.386419 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:03:55.386383 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:00.102693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:00.102662 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:01.573525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:01.573481 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:01.999770 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:01.999731 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:02.682094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:02.681719 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:04:03.262004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:03.261969 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:03.486711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:03.486672 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:05.355417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:05.355374 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:07.224368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:07.224330 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:07.419556 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:07.419524 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:07.577769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:07.577687 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:08.821646 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:08.821622 4285 generic.go:356] "Generic (PLEG): container finished" podID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerID="c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def" exitCode=0 Nov 02 22:04:08.821975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:08.821694 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" event={"ID":"edaaab34-eddb-4c62-b119-b51fa28512c9","Type":"ContainerDied","Data":"c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def"} Nov 02 22:04:08.822071 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:08.822056 4285 scope.go:122] "RemoveContainer" containerID="c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def" Nov 02 22:04:08.823677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:08.823649 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:08.823759 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:08.823718 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:08.823759 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:08.823753 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:09.848835 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:09.848797 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" event={"ID":"edaaab34-eddb-4c62-b119-b51fa28512c9","Type":"ContainerStarted","Data":"23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc"} Nov 02 22:04:10.082413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.082383 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:04:10.082618 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.082594 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 22:04:10.082694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.082651 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 22:04:10.286502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.286463 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:10.484565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.484538 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:10.833430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.833395 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:11.010211 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:10.997770 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8"] Nov 02 22:04:11.246151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.086155 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 22:04:11.246444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.246412 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 22:04:11.270154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.260248 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.352553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.352507 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.358819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.358787 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82frv\" (UniqueName: \"kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv\") pod \"payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.402501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.388375 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3eb190-dockercfg-r5bdc\"" Nov 02 22:04:11.402501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.388436 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3eb190\"" Nov 02 22:04:11.411145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.404231 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8"] Nov 02 22:04:11.505085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.504989 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.505085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.505057 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82frv\" (UniqueName: \"kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv\") pod \"payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.505890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.505864 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.562706 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.562670 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82frv\" (UniqueName: \"kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv\") pod \"payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:11.704341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:11.704302 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:12.082077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.081992 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 22:04:12.082451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.082069 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 22:04:12.149286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.149259 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8"] Nov 02 22:04:12.152509 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:04:12.152482 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72063619_3210_4799_a2ea_4f4bdd251d82.slice/crio-f18949878acfcbedf90d8ab0331978acd035333c4880fadfc55e6f29ca9eddfc WatchSource:0}: Error finding container f18949878acfcbedf90d8ab0331978acd035333c4880fadfc55e6f29ca9eddfc: Status 404 returned error can't find the container with id f18949878acfcbedf90d8ab0331978acd035333c4880fadfc55e6f29ca9eddfc Nov 02 22:04:12.249839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.249765 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:12.399417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.399337 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:12.563244 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.563205 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:12.970284 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.970242 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" event={"ID":"72063619-3210-4799-a2ea-4f4bdd251d82","Type":"ContainerStarted","Data":"17614e22ac9f829e98694087af15973a9686cfbcfa31f27854d1fcee5ad17dc0"} Nov 02 22:04:12.970436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.970300 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" event={"ID":"72063619-3210-4799-a2ea-4f4bdd251d82","Type":"ContainerStarted","Data":"f18949878acfcbedf90d8ab0331978acd035333c4880fadfc55e6f29ca9eddfc"} Nov 02 22:04:12.970436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:12.970424 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:13.082273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:13.082237 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 22:04:13.082684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:13.082284 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 22:04:13.086596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:13.086552 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" podStartSLOduration=3.086539121 podStartE2EDuration="3.086539121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:13.082338114 +0000 UTC m=+1792.043888434" watchObservedRunningTime="2026-11-02 22:04:13.086539121 +0000 UTC m=+1792.048089439" Nov 02 22:04:14.082899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:14.082865 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 22:04:14.083319 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:14.082912 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 22:04:15.019460 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:15.019426 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:15.215596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:15.215559 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:04:15.216443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:15.216382 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:04:15.219283 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:15.219264 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:04:16.008293 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:04:16.008259 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:04:19.652478 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:19.652442 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:23.687333 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:23.687298 4285 scope.go:122] "RemoveContainer" containerID="4a4f39299688a8cffd3e50f23b918a9372c876a813397d0cfb598b715c9fa8a2" Nov 02 22:04:23.980279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:23.980246 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:04:24.438241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:24.438039 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:29.110676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:29.110633 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:33.431139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.431085 4285 generic.go:356] "Generic (PLEG): container finished" podID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerID="4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4" exitCode=0 Nov 02 22:04:33.431585 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.431158 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" event={"ID":"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10","Type":"ContainerDied","Data":"4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4"} Nov 02 22:04:33.431766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.431748 4285 scope.go:122] "RemoveContainer" containerID="4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4" Nov 02 22:04:33.433701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.433669 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:33.433823 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.433744 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:33.433823 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.433777 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:33.831074 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:33.830990 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:34.458422 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.458327 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" event={"ID":"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10","Type":"ContainerStarted","Data":"c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536"} Nov 02 22:04:34.461684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.461649 4285 generic.go:356] "Generic (PLEG): container finished" podID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerID="1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b" exitCode=0 Nov 02 22:04:34.461884 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.461739 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" event={"ID":"5d5e7c02-59cc-4491-8d41-567112f067aa","Type":"ContainerDied","Data":"1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b"} Nov 02 22:04:34.462309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.462292 4285 scope.go:122] "RemoveContainer" containerID="1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b" Nov 02 22:04:34.464297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.464269 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:34.464399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.464352 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:34.464468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.464402 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:34.562101 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.562030 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:04:34.563326 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.563301 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:34.563468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:34.563353 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:35.473203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:35.473070 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" event={"ID":"5d5e7c02-59cc-4491-8d41-567112f067aa","Type":"ContainerStarted","Data":"fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a"} Nov 02 22:04:35.535528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:35.535490 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:04:35.548153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:35.547278 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:35.548153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:35.547346 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:35.569621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:35.569580 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:35.569919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:35.569888 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:36.193436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:36.193398 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:36.536593 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:36.536494 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:36.537188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:36.537156 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:36.574955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:36.574915 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:36.575414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:36.575384 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:37.015155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:36.954848 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:04:37.173185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.164405 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:37.344861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.344722 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:37.345833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.345806 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:04:37.544775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.543279 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:37.544775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.543345 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:37.568354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.568232 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:37.568354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:37.568292 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:38.498439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:38.498407 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:38.534716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:38.534675 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:38.535002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:38.534967 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:38.562541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:38.562504 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:38.563096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:38.563062 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:39.542149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:39.541681 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:39.542149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:39.541745 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:39.565151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:39.563327 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:39.565151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:39.563384 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:40.537521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:40.537483 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:40.577294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:40.577248 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:40.578300 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:40.573339 4285 patch_prober.go:29] interesting pod/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 02 22:04:40.578499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:40.578475 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 02 22:04:41.411162 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.411132 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:41.537520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.537483 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:04:41.537831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.537800 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 22:04:41.568240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.568205 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:04:41.569338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.569316 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:04:41.596470 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.596437 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:41.825221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:41.825177 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:42.538873 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:42.538842 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:04:42.539861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:42.539834 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:04:42.684724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:42.684689 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:42.863197 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:42.863048 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:43.738198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:43.738107 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:46.008383 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:04:46.008347 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:04:47.887876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:47.887834 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:48.196471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.196429 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:48.354903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.354834 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:48.550202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.544846 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j"] Nov 02 22:04:48.644950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.644913 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:48.661357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.661326 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j"] Nov 02 22:04:48.661722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.661600 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.714357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.714326 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-224f53\"" Nov 02 22:04:48.745837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.745803 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rwz\" (UniqueName: \"kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz\") pod \"payload-processing-e2e-api-a-224f53-5967d66c9-67z4j\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.746045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.745878 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-224f53-5967d66c9-67z4j\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.763715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.763687 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-224f53-dockercfg-5tnm4\"" Nov 02 22:04:48.847265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.847179 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rwz\" (UniqueName: \"kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz\") pod \"payload-processing-e2e-api-a-224f53-5967d66c9-67z4j\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.847467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.847277 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-224f53-5967d66c9-67z4j\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.848004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.847984 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-224f53-5967d66c9-67z4j\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.903308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.903278 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rwz\" (UniqueName: \"kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz\") pod \"payload-processing-e2e-api-a-224f53-5967d66c9-67z4j\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:48.916088 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.916023 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:48.978269 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:48.978236 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:49.458156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:49.447280 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:49.599281 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:49.599225 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j"] Nov 02 22:04:49.652652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:49.652614 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:49.812503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:49.812462 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" event={"ID":"b9bc72b6-9d26-4914-a5f8-69364c340da5","Type":"ContainerStarted","Data":"9be5165dd710349e878bfb448a60f91b11040ee276791fef398155a46f09bf40"} Nov 02 22:04:50.004542 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:50.004342 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:50.820885 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:50.820850 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" event={"ID":"b9bc72b6-9d26-4914-a5f8-69364c340da5","Type":"ContainerStarted","Data":"23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1"} Nov 02 22:04:50.821086 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:50.821015 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:04:50.987381 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:50.987326 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" podStartSLOduration=2.987308943 podStartE2EDuration="2.987308943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:50.932818544 +0000 UTC m=+1829.894368862" watchObservedRunningTime="2026-11-02 22:04:50.987308943 +0000 UTC m=+1829.948859265" Nov 02 22:04:51.768140 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:51.768097 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856"] Nov 02 22:04:52.037692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:52.037614 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:52.216878 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:52.216836 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:53.818637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:53.818599 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:53.994537 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:53.994498 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:54.181175 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:54.181040 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:54.352383 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:54.352350 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:56.769055 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:56.769014 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerName="payload-pre-processing" containerID="cri-o://61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76" gracePeriod=25 Nov 02 22:04:57.900691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:57.900657 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:57.990165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:57.990118 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.163650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.163571 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.597220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.574045 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz"] Nov 02 22:04:58.619520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.619387 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:58.619766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.619406 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz"] Nov 02 22:04:58.762268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.762242 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-587629\"" Nov 02 22:04:58.762595 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.762572 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-587629-dockercfg-rn897\"" Nov 02 22:04:58.830601 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.830573 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xhf\" (UniqueName: \"kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf\") pod \"payload-processing-e2e-api-b-587629-589679fb55-9vhrz\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:58.830790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.830717 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume\") pod \"payload-processing-e2e-api-b-587629-589679fb55-9vhrz\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:58.932098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.932013 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume\") pod \"payload-processing-e2e-api-b-587629-589679fb55-9vhrz\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:58.932602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.932229 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xhf\" (UniqueName: \"kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf\") pod \"payload-processing-e2e-api-b-587629-589679fb55-9vhrz\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:58.932871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.932842 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume\") pod \"payload-processing-e2e-api-b-587629-589679fb55-9vhrz\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:58.990520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:58.990490 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xhf\" (UniqueName: \"kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf\") pod \"payload-processing-e2e-api-b-587629-589679fb55-9vhrz\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:59.063865 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:59.063837 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:04:59.422505 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:59.422469 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:59.495306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:59.495251 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz"] Nov 02 22:04:59.891439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:59.891394 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" event={"ID":"e8e87085-4bd5-42cf-8599-1a916c6bb3be","Type":"ContainerStarted","Data":"44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157"} Nov 02 22:04:59.891626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:59.891448 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" event={"ID":"e8e87085-4bd5-42cf-8599-1a916c6bb3be","Type":"ContainerStarted","Data":"d3c4bbb3faa40f19cf835861f506d3f281ec607a62843e9e7b0fc5e38b750fe6"} Nov 02 22:04:59.891626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:04:59.891487 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:05:00.005381 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:00.005326 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" podStartSLOduration=2.005311275 podStartE2EDuration="2.005311275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:00.002403804 +0000 UTC m=+1838.963954128" watchObservedRunningTime="2026-11-02 22:05:00.005311275 +0000 UTC m=+1838.966861598" Nov 02 22:05:01.832615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:01.832577 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:05:03.779200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:03.775730 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:04.033259 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.033175 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x"] Nov 02 22:05:04.037274 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.037253 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.044154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.044106 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.044292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.044265 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxtn\" (UniqueName: \"kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.044425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.044403 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.044592 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.044574 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.093345 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.093305 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x"] Nov 02 22:05:04.137548 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.137506 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-267d89-serving-cert\"" Nov 02 22:05:04.148036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.147992 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.148279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.148088 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.148279 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:05:04.148149 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-serving-cert: secret "maas-api-e2e-shared-b-w4-267d89-serving-cert" not found Nov 02 22:05:04.148279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.148197 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxtn\" (UniqueName: \"kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.148279 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:05:04.148219 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls podName:eb47c9e0-ddf8-4953-ade9-1d2bfeeec869 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:04.648199966 +0000 UTC m=+1843.609750275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls") pod "maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" (UID: "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869") : secret "maas-api-e2e-shared-b-w4-267d89-serving-cert" not found Nov 02 22:05:04.148509 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.148338 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.149096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.149071 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.150696 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.150677 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.204325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.204292 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxtn\" (UniqueName: \"kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.248455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.248415 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:04.403031 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.402955 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:04.552912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.552874 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:04.664602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.664522 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.666950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.666932 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:04.951430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:04.951398 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:05.333558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:05.333526 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x"] Nov 02 22:05:05.334695 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:05:05.334662 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb47c9e0_ddf8_4953_ade9_1d2bfeeec869.slice/crio-5b92fed6222f1265cecd594c8bc42e45c68abf7e2cf6d5f6b82633e4ef5e378a WatchSource:0}: Error finding container 5b92fed6222f1265cecd594c8bc42e45c68abf7e2cf6d5f6b82633e4ef5e378a: Status 404 returned error can't find the container with id 5b92fed6222f1265cecd594c8bc42e45c68abf7e2cf6d5f6b82633e4ef5e378a Nov 02 22:05:05.941325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:05.941288 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" event={"ID":"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869","Type":"ContainerStarted","Data":"5b92fed6222f1265cecd594c8bc42e45c68abf7e2cf6d5f6b82633e4ef5e378a"} Nov 02 22:05:06.951751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:06.951713 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" event={"ID":"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869","Type":"ContainerStarted","Data":"32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a"} Nov 02 22:05:06.952444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:06.952013 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:07.068167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:07.068091 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" podStartSLOduration=3.218771627 podStartE2EDuration="4.068077586s" totalImagesPullingTime="849.305959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:07.063857803 +0000 UTC m=+1846.025408124" watchObservedRunningTime="2026-11-02 22:05:07.068077586 +0000 UTC m=+1846.029627905" Nov 02 22:05:07.204580 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:07.204498 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:07.373146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:07.372423 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:07.900511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:07.900466 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:05:08.620171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:08.620137 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:08.779034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:08.778990 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:09.441083 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:09.441047 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:10.902096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:10.902069 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:05:12.717343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:12.717299 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:12.866154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:12.866093 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:12.968429 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:12.968361 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:05:14.678753 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:14.678715 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:16.009900 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:05:16.009854 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:05:17.067028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.067003 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 22:05:17.085906 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.085875 4285 generic.go:356] "Generic (PLEG): container finished" podID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerID="61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76" exitCode=0 Nov 02 22:05:17.086049 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.085971 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" Nov 02 22:05:17.086049 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.086028 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" event={"ID":"2c8d9759-e788-48f0-a6d2-ec9b88271cd1","Type":"ContainerDied","Data":"61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76"} Nov 02 22:05:17.086217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.086092 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856" event={"ID":"2c8d9759-e788-48f0-a6d2-ec9b88271cd1","Type":"ContainerDied","Data":"bc3186eb7436cd1f128c678316a50a77dbbeaf39444e86a68d41c76736ae12ba"} Nov 02 22:05:17.086217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.086145 4285 scope.go:122] "RemoveContainer" containerID="61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76" Nov 02 22:05:17.097939 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.097922 4285 scope.go:122] "RemoveContainer" containerID="61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76" Nov 02 22:05:17.098281 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:05:17.098255 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76\": container with ID starting with 61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76 not found: ID does not exist" containerID="61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76" Nov 02 22:05:17.098406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.098282 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76"} err="failed to get container status \"61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76\": rpc error: code = NotFound desc = could not find container \"61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76\": container with ID starting with 61ea4f65049c12c7a1ddb7634db61cebb1d4e962e59f5e3ddbe39a778f945d76 not found: ID does not exist" Nov 02 22:05:17.189050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.189022 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume\") pod \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " Nov 02 22:05:17.189303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.189283 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77\" (UniqueName: \"kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77\") pod \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\" (UID: \"2c8d9759-e788-48f0-a6d2-ec9b88271cd1\") " Nov 02 22:05:17.189400 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.189373 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume" pod "2c8d9759-e788-48f0-a6d2-ec9b88271cd1" (UID: "2c8d9759-e788-48f0-a6d2-ec9b88271cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:05:17.189802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.189781 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:17.191372 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.191348 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77" pod "2c8d9759-e788-48f0-a6d2-ec9b88271cd1" (UID: "2c8d9759-e788-48f0-a6d2-ec9b88271cd1"). InnerVolumeSpecName "kube-api-access-kvn77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:17.291427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.291343 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvn77\" (UniqueName: \"kubernetes.io/projected/2c8d9759-e788-48f0-a6d2-ec9b88271cd1-kube-api-access-kvn77\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:05:17.508477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.508439 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856"] Nov 02 22:05:17.512195 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:17.512171 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2be7cc3-598d7c6487-2g856"] Nov 02 22:05:19.175405 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:19.175374 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" path="/var/lib/kubelet/pods/2c8d9759-e788-48f0-a6d2-ec9b88271cd1/volumes" Nov 02 22:05:19.339552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:19.339510 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:19.540579 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:19.540543 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:19.827267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:19.827174 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:24.062465 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:24.062380 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:28.603884 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:28.603844 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:28.770471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:28.770439 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:28.926242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:28.924648 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:33.439081 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:33.439048 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.132749 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:38.132703 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:41.143195 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:41.143151 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:42.777900 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:42.777866 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:46.009664 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:05:46.009623 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:05:46.823325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:46.823282 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:47.484864 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:47.484835 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:52.336430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:52.336386 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:55.260288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:55.260239 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-przjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:05:55.290631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:55.269233 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:05:58.026547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:05:58.026433 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.844551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:02.844512 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:07.514461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:07.514424 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:07.718812 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:07.718768 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:07.886202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:07.886100 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:12.514647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:12.514603 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:13.029654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:13.029590 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:16.007089 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:06:16.007051 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:06:17.228788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:17.228749 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:21.732519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.732484 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh"] Nov 02 22:06:21.733302 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.733285 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerName="payload-pre-processing" Nov 02 22:06:21.733364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.733302 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerName="payload-pre-processing" Nov 02 22:06:21.733425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.733416 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8d9759-e788-48f0-a6d2-ec9b88271cd1" containerName="payload-pre-processing" Nov 02 22:06:21.737275 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.737257 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.805337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.805305 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.805517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.805421 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklg4\" (UniqueName: \"kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.805517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.805476 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.805653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.805602 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.865701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.865666 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh"] Nov 02 22:06:21.888622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.888588 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-587629-serving-cert\"" Nov 02 22:06:21.907669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.907611 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklg4\" (UniqueName: \"kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.907870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.907699 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.907870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.907775 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.908072 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.908005 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.908367 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:06:21.908340 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-587629-serving-cert: secret "maas-api-e2e-api-b-587629-serving-cert" not found Nov 02 22:06:21.908473 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:06:21.908440 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls podName:9090c019-7e03-4f72-90e1-2969de759f29 nodeName:}" failed. No retries permitted until 2026-11-02 22:06:22.408416622 +0000 UTC m=+1921.369966922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls") pod "maas-api-e2e-api-b-587629-767bfffd59-r6pzh" (UID: "9090c019-7e03-4f72-90e1-2969de759f29") : secret "maas-api-e2e-api-b-587629-serving-cert" not found Nov 02 22:06:21.909253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.909224 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.910866 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.910842 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:21.946756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.946724 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:21.965160 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:21.965101 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklg4\" (UniqueName: \"kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:22.113679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:22.113595 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:22.264558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:22.264528 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:22.444371 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:22.444337 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:22.447030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:22.447000 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\") pod \"maas-api-e2e-api-b-587629-767bfffd59-r6pzh\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:22.653354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:22.653070 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:23.031525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:23.031445 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh"] Nov 02 22:06:23.036048 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:06:23.036004 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9090c019_7e03_4f72_90e1_2969de759f29.slice/crio-314b9b1926ad1ce8c32fe698b503ad1e2318a7cff023ca6349593cf08f6e8c7e WatchSource:0}: Error finding container 314b9b1926ad1ce8c32fe698b503ad1e2318a7cff023ca6349593cf08f6e8c7e: Status 404 returned error can't find the container with id 314b9b1926ad1ce8c32fe698b503ad1e2318a7cff023ca6349593cf08f6e8c7e Nov 02 22:06:23.886760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:23.886723 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" event={"ID":"9090c019-7e03-4f72-90e1-2969de759f29","Type":"ContainerStarted","Data":"314b9b1926ad1ce8c32fe698b503ad1e2318a7cff023ca6349593cf08f6e8c7e"} Nov 02 22:06:24.895467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:24.895429 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" event={"ID":"9090c019-7e03-4f72-90e1-2969de759f29","Type":"ContainerStarted","Data":"0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70"} Nov 02 22:06:24.895860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:24.895522 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:25.010004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:25.009952 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" podStartSLOduration=3.11955388 podStartE2EDuration="4.009937713s" totalImagesPullingTime="890.383833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:25.008271449 +0000 UTC m=+1923.969821783" watchObservedRunningTime="2026-11-02 22:06:25.009937713 +0000 UTC m=+1923.971488032" Nov 02 22:06:26.800797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:26.800763 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:30.908095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:30.908064 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:06:31.515484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:31.515441 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:36.210159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:36.210049 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:41.104785 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:41.104729 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:42.132871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:42.132802 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:46.008841 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:06:46.008805 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:06:46.114045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:46.114010 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:48.575015 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:48.574976 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:50.223338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:50.223294 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:51.807382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:51.807317 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:52.123536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:52.123445 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:52.279680 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:52.279645 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:54.376616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:54.376559 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:55.229677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:55.229639 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:55.802165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:55.801020 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:56.219436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:56.219404 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:56.543137 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:56.543047 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:57.224779 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:57.224713 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:59.689218 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:59.689177 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:59.914774 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:06:59.914741 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:01.732249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:01.732219 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:01.904220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:01.904191 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:02.073447 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:02.073356 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:02.262629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:02.262594 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:02.423474 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:02.423390 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:02.579610 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:02.579572 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:07.276571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:07.276527 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:07.607904 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:07.607799 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:07.805575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:07.805542 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:11.534973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:11.534916 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:12.210159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:12.161050 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:12.717199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:12.717149 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:12.915312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:12.915253 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:13.379690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:13.379655 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:16.009222 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:16.009182 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:07:16.415991 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:16.415911 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:16.585342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:16.585313 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:16.769646 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:16.769610 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:18.202393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:18.202361 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:19.634920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:19.634887 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:19.863775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:19.863736 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:20.103047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.102995 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt"] Nov 02 22:07:20.108577 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.108546 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt"] Nov 02 22:07:20.108745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.108643 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.365235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.260321 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2050f3\"" Nov 02 22:07:20.365235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.260398 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2050f3-dockercfg-ggr7c\"" Nov 02 22:07:20.365235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.266068 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx5kr\" (UniqueName: \"kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr\") pod \"payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.365235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.266441 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.385070 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.384943 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.385807 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.385783 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx5kr\" (UniqueName: \"kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr\") pod \"payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.386345 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.386112 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.478955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.465178 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87"] Nov 02 22:07:20.485922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.485896 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87"] Nov 02 22:07:20.486093 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.486081 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.486610 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.486583 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx5kr\" (UniqueName: \"kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr\") pod \"payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.588871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.588840 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtvt2\" (UniqueName: \"kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2\") pod \"payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.589044 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.589013 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.690176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.690042 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.690648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.690283 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtvt2\" (UniqueName: \"kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2\") pod \"payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.690810 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.690788 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.729376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.729341 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:20.746611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.746585 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtvt2\" (UniqueName: \"kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2\") pod \"payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:20.826444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:20.826406 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:21.118325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.118299 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt"] Nov 02 22:07:21.120459 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:07:21.120430 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1198c370_1272_41a0_9d26_2d0bcfb3495c.slice/crio-01b68d65fe3bf8e9380fb95c6a32527e7b497acbd62bad51c08a46f8e4269e0a WatchSource:0}: Error finding container 01b68d65fe3bf8e9380fb95c6a32527e7b497acbd62bad51c08a46f8e4269e0a: Status 404 returned error can't find the container with id 01b68d65fe3bf8e9380fb95c6a32527e7b497acbd62bad51c08a46f8e4269e0a Nov 02 22:07:21.227582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.227557 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87"] Nov 02 22:07:21.229159 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:07:21.229115 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a6b374_ffca_44de_ad31_971cff1de3fe.slice/crio-c5c02b47fc36b1e2d6b24c691166045fd7b4e37948e4d097fe0c931fbfd0b291 WatchSource:0}: Error finding container c5c02b47fc36b1e2d6b24c691166045fd7b4e37948e4d097fe0c931fbfd0b291: Status 404 returned error can't find the container with id c5c02b47fc36b1e2d6b24c691166045fd7b4e37948e4d097fe0c931fbfd0b291 Nov 02 22:07:21.393629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.393544 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:21.702355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.702314 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" event={"ID":"1198c370-1272-41a0-9d26-2d0bcfb3495c","Type":"ContainerStarted","Data":"7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33"} Nov 02 22:07:21.702890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.702383 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" event={"ID":"1198c370-1272-41a0-9d26-2d0bcfb3495c","Type":"ContainerStarted","Data":"01b68d65fe3bf8e9380fb95c6a32527e7b497acbd62bad51c08a46f8e4269e0a"} Nov 02 22:07:21.702890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.702412 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:21.704206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.704178 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" event={"ID":"49a6b374-ffca-44de-ad31-971cff1de3fe","Type":"ContainerStarted","Data":"40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19"} Nov 02 22:07:21.704354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.704234 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:21.704354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.704259 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" event={"ID":"49a6b374-ffca-44de-ad31-971cff1de3fe","Type":"ContainerStarted","Data":"c5c02b47fc36b1e2d6b24c691166045fd7b4e37948e4d097fe0c931fbfd0b291"} Nov 02 22:07:21.818392 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.818338 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" podStartSLOduration=2.818320769 podStartE2EDuration="2.818320769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:21.813964078 +0000 UTC m=+1980.775514399" watchObservedRunningTime="2026-11-02 22:07:21.818320769 +0000 UTC m=+1980.779871090" Nov 02 22:07:21.943389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:21.938176 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" podStartSLOduration=1.9381137910000001 podStartE2EDuration="1.938113791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:21.936609544 +0000 UTC m=+1980.898159865" watchObservedRunningTime="2026-11-02 22:07:21.938113791 +0000 UTC m=+1980.899664112" Nov 02 22:07:22.766831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:22.766794 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:22.928729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:22.928677 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:23.154209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:23.154091 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x"] Nov 02 22:07:23.265794 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:23.265763 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj"] Nov 02 22:07:23.329822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:23.329787 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:23.547871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:23.547838 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:23.724976 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:23.724753 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:28.155199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.155117 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" podUID="6aa9aecd-6b68-459f-82f5-a32118a08a0b" containerName="payload-processing" containerID="cri-o://5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99" gracePeriod=25 Nov 02 22:07:28.266362 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.266322 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" podUID="9b4043e2-d29b-44e4-844b-5fc0a2d9201a" containerName="payload-pre-processing" containerID="cri-o://ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248" gracePeriod=25 Nov 02 22:07:28.400530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.400465 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:28.519468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.519067 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:07:28.557966 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.557916 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:28.673500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.673459 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:07:28.708516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.708483 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume\") pod \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " Nov 02 22:07:28.708672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.708630 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g\" (UniqueName: \"kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g\") pod \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\" (UID: \"6aa9aecd-6b68-459f-82f5-a32118a08a0b\") " Nov 02 22:07:28.708908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.708882 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume" pod "6aa9aecd-6b68-459f-82f5-a32118a08a0b" (UID: "6aa9aecd-6b68-459f-82f5-a32118a08a0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:28.709396 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.709374 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa9aecd-6b68-459f-82f5-a32118a08a0b-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:28.710994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.710964 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g" pod "6aa9aecd-6b68-459f-82f5-a32118a08a0b" (UID: "6aa9aecd-6b68-459f-82f5-a32118a08a0b"). InnerVolumeSpecName "kube-api-access-z826g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:28.810240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.810142 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz\" (UniqueName: \"kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz\") pod \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " Nov 02 22:07:28.810240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.810236 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume\") pod \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\" (UID: \"9b4043e2-d29b-44e4-844b-5fc0a2d9201a\") " Nov 02 22:07:28.810673 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.810643 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume" pod "9b4043e2-d29b-44e4-844b-5fc0a2d9201a" (UID: "9b4043e2-d29b-44e4-844b-5fc0a2d9201a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:28.811366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.811336 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:28.811468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.811371 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z826g\" (UniqueName: \"kubernetes.io/projected/6aa9aecd-6b68-459f-82f5-a32118a08a0b-kube-api-access-z826g\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:28.812332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.812304 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz" pod "9b4043e2-d29b-44e4-844b-5fc0a2d9201a" (UID: "9b4043e2-d29b-44e4-844b-5fc0a2d9201a"). InnerVolumeSpecName "kube-api-access-k9vhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:28.913265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.913224 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vhz\" (UniqueName: \"kubernetes.io/projected/9b4043e2-d29b-44e4-844b-5fc0a2d9201a-kube-api-access-k9vhz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:28.940828 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.940794 4285 generic.go:356] "Generic (PLEG): container finished" podID="6aa9aecd-6b68-459f-82f5-a32118a08a0b" containerID="5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99" exitCode=0 Nov 02 22:07:28.941023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.940837 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" Nov 02 22:07:28.941023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.940882 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" event={"ID":"6aa9aecd-6b68-459f-82f5-a32118a08a0b","Type":"ContainerDied","Data":"5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99"} Nov 02 22:07:28.941023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.940937 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x" event={"ID":"6aa9aecd-6b68-459f-82f5-a32118a08a0b","Type":"ContainerDied","Data":"b5d80b4d9608fadf114067cbf80b4400c371ebdd3ea5d932d85e6438628ee16e"} Nov 02 22:07:28.941023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.940967 4285 scope.go:122] "RemoveContainer" containerID="5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99" Nov 02 22:07:28.942774 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.942748 4285 generic.go:356] "Generic (PLEG): container finished" podID="9b4043e2-d29b-44e4-844b-5fc0a2d9201a" containerID="ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248" exitCode=0 Nov 02 22:07:28.942892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.942790 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" Nov 02 22:07:28.942892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.942833 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" event={"ID":"9b4043e2-d29b-44e4-844b-5fc0a2d9201a","Type":"ContainerDied","Data":"ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248"} Nov 02 22:07:28.942892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.942864 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj" event={"ID":"9b4043e2-d29b-44e4-844b-5fc0a2d9201a","Type":"ContainerDied","Data":"4acfd40a7439e58331babf2c7404378342d17c871fdc977b752468738f476cab"} Nov 02 22:07:28.955107 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.955088 4285 scope.go:122] "RemoveContainer" containerID="5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99" Nov 02 22:07:28.955742 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:28.955721 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99\": container with ID starting with 5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99 not found: ID does not exist" containerID="5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99" Nov 02 22:07:28.955855 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.955750 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99"} err="failed to get container status \"5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99\": rpc error: code = NotFound desc = could not find container \"5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99\": container with ID starting with 5a7eed6673ef16937c87b833fde1683f8e6184397647b2ad2111c29f7f74cb99 not found: ID does not exist" Nov 02 22:07:28.955855 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.955766 4285 scope.go:122] "RemoveContainer" containerID="ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248" Nov 02 22:07:28.965463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.965445 4285 scope.go:122] "RemoveContainer" containerID="ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248" Nov 02 22:07:28.965741 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:28.965721 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248\": container with ID starting with ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248 not found: ID does not exist" containerID="ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248" Nov 02 22:07:28.965800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:28.965749 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248"} err="failed to get container status \"ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248\": rpc error: code = NotFound desc = could not find container \"ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248\": container with ID starting with ece3bf8864c8ccca7db31d054852f1a786d6deba5799a87175331d211bb86248 not found: ID does not exist" Nov 02 22:07:29.105520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:29.105440 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj"] Nov 02 22:07:29.109530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:29.109506 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-204000-544459645f-fkbgj"] Nov 02 22:07:29.175833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:29.175806 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4043e2-d29b-44e4-844b-5fc0a2d9201a" path="/var/lib/kubelet/pods/9b4043e2-d29b-44e4-844b-5fc0a2d9201a/volumes" Nov 02 22:07:29.215895 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:29.215866 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x"] Nov 02 22:07:29.221088 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:29.221060 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-204000-5585f48cb5-bfb9x"] Nov 02 22:07:31.175555 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:31.175525 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa9aecd-6b68-459f-82f5-a32118a08a0b" path="/var/lib/kubelet/pods/6aa9aecd-6b68-459f-82f5-a32118a08a0b/volumes" Nov 02 22:07:32.715437 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:32.715398 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:07:32.715933 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:32.715484 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:07:33.452150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:33.450395 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:34.157667 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:34.157604 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:38.217640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:38.217608 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:42.840566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:42.840529 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:45.685936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.685900 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc"] Nov 02 22:07:45.686458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.686160 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" containerID="cri-o://c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af" gracePeriod=30 Nov 02 22:07:45.735597 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.735568 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx"] Nov 02 22:07:45.735782 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.735763 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" containerID="cri-o://f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875" gracePeriod=30 Nov 02 22:07:45.824519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.824482 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p"] Nov 02 22:07:45.824751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.824724 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" containerID="cri-o://056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3" gracePeriod=30 Nov 02 22:07:45.839398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.836184 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl"] Nov 02 22:07:45.839398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:45.836411 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" containerID="cri-o://d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf" gracePeriod=30 Nov 02 22:07:46.007778 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:46.007744 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:07:47.744375 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:47.744341 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:49.636882 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:49.636844 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8"] Nov 02 22:07:49.637450 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:49.637075 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" containerID="cri-o://2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2" gracePeriod=30 Nov 02 22:07:50.008772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:50.008736 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:07:50.008934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:50.008802 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:07:50.366938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:50.366856 4285 patch_prober.go:29] interesting pod/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:07:50.366938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:50.366913 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:07:50.522380 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:50.522348 4285 patch_prober.go:29] interesting pod/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:07:50.522537 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:50.522429 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:07:51.045831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.045288 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 22:07:51.119170 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.119064 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.175621 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.175673 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.175749 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.175811 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.175970 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.176036 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.176193 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.176230 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.176356 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket\") pod \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\" (UID: \"d7b382c8-0005-4373-8a93-cd6d2ed4212a\") " Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.177875 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.178066 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.178540 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.179145 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.180638 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.182415 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.185096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.182644 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.186183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.185677 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.187872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.187834 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj" pod "d7b382c8-0005-4373-8a93-cd6d2ed4212a" (UID: "d7b382c8-0005-4373-8a93-cd6d2ed4212a"). InnerVolumeSpecName "kube-api-access-hvsvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.192203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.192176 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 22:07:51.217012 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.216949 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 22:07:51.229139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.229094 4285 generic.go:356] "Generic (PLEG): container finished" podID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerID="c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af" exitCode=0 Nov 02 22:07:51.229284 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.229249 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" event={"ID":"d7b382c8-0005-4373-8a93-cd6d2ed4212a","Type":"ContainerDied","Data":"c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af"} Nov 02 22:07:51.229354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.229308 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" event={"ID":"d7b382c8-0005-4373-8a93-cd6d2ed4212a","Type":"ContainerDied","Data":"4e5ee9b4c51eb6cd05db7eb252189077907fb7c9d240c024eef4393c930aed54"} Nov 02 22:07:51.229354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.229339 4285 scope.go:122] "RemoveContainer" containerID="c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af" Nov 02 22:07:51.229504 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.229487 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc" Nov 02 22:07:51.236237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.236110 4285 generic.go:356] "Generic (PLEG): container finished" podID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerID="d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf" exitCode=0 Nov 02 22:07:51.236340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.236155 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" event={"ID":"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a","Type":"ContainerDied","Data":"d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf"} Nov 02 22:07:51.236406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.236196 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" Nov 02 22:07:51.236406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.236366 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl" event={"ID":"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a","Type":"ContainerDied","Data":"2744b57ea057948eda5e9729feb0556d6c655ae8f144fca1994b8fe0960a8928"} Nov 02 22:07:51.237889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.237867 4285 generic.go:356] "Generic (PLEG): container finished" podID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerID="056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3" exitCode=0 Nov 02 22:07:51.237989 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.237910 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" Nov 02 22:07:51.237989 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.237955 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" event={"ID":"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43","Type":"ContainerDied","Data":"056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3"} Nov 02 22:07:51.238111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.238007 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p" event={"ID":"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43","Type":"ContainerDied","Data":"d41dfe5cf7e6bd6acea784f793f791daf1b4b6febb03d6cd2308e1e7fd6eafa3"} Nov 02 22:07:51.239501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.239482 4285 generic.go:356] "Generic (PLEG): container finished" podID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerID="f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875" exitCode=0 Nov 02 22:07:51.239596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.239532 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" event={"ID":"cbe5f8f9-9acd-4919-9b79-213f822f8dbf","Type":"ContainerDied","Data":"f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875"} Nov 02 22:07:51.239596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.239541 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" Nov 02 22:07:51.239596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.239559 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx" event={"ID":"cbe5f8f9-9acd-4919-9b79-213f822f8dbf","Type":"ContainerDied","Data":"8f832c8c756c346a2216e84ab3924ad2f24614a0b768186fcaa10d85c2a4c808"} Nov 02 22:07:51.267309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.265868 4285 scope.go:122] "RemoveContainer" containerID="c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af" Nov 02 22:07:51.267309 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:51.267015 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af\": container with ID starting with c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af not found: ID does not exist" containerID="c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af" Nov 02 22:07:51.267309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.267051 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af"} err="failed to get container status \"c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af\": rpc error: code = NotFound desc = could not find container \"c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af\": container with ID starting with c519ea4bb180254c53d5f40cc94cbb9c08e29f9ad0388eaabe3651f0d9a0d0af not found: ID does not exist" Nov 02 22:07:51.267309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.267074 4285 scope.go:122] "RemoveContainer" containerID="d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf" Nov 02 22:07:51.278357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278329 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.278521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278417 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.278521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278469 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.278521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278513 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278541 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278584 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278617 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278655 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278716 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278755 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.278821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278804 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278835 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278874 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278911 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278961 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278990 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279086 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279192 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279235 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.279363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279286 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279378 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279454 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279481 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279523 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket\") pod \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\" (UID: \"9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279552 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279620 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket\") pod \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\" (UID: \"a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43\") " Nov 02 22:07:51.279841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279656 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket\") pod \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\" (UID: \"cbe5f8f9-9acd-4919-9b79-213f822f8dbf\") " Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280605 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280631 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280646 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280661 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280675 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b382c8-0005-4373-8a93-cd6d2ed4212a-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280689 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvsvj\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-kube-api-access-hvsvj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280706 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280720 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.280891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280736 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b382c8-0005-4373-8a93-cd6d2ed4212a-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.281740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.278956 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.281740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.279526 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.281740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280978 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.281740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.280998 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.281740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.281071 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.281740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.281434 4285 scope.go:122] "RemoveContainer" containerID="d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf" Nov 02 22:07:51.282246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.281745 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.282246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.281961 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.282246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.282091 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.282499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.282477 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.291109 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.290971 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.291512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.291281 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.291512 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:51.291373 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf\": container with ID starting with d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf not found: ID does not exist" containerID="d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf" Nov 02 22:07:51.291512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.291405 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf"} err="failed to get container status \"d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf\": rpc error: code = NotFound desc = could not find container \"d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf\": container with ID starting with d80645ac5fa45ab928340c6ce9f2c2c154492e6544228f76b119fdebc54e0fcf not found: ID does not exist" Nov 02 22:07:51.291512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.291428 4285 scope.go:122] "RemoveContainer" containerID="056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3" Nov 02 22:07:51.292542 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.292518 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.293586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.293555 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.294498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.294473 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:51.294833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.294790 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.305929 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.305895 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.310727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.310697 4285 scope.go:122] "RemoveContainer" containerID="056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3" Nov 02 22:07:51.311109 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:51.311061 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3\": container with ID starting with 056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3 not found: ID does not exist" containerID="056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3" Nov 02 22:07:51.311109 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.311097 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3"} err="failed to get container status \"056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3\": rpc error: code = NotFound desc = could not find container \"056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3\": container with ID starting with 056102b7525be9ae7ef3c391d7d333c5ec3f384be116b582fae6d7c46801a8f3 not found: ID does not exist" Nov 02 22:07:51.311306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.311137 4285 scope.go:122] "RemoveContainer" containerID="f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875" Nov 02 22:07:51.312077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.311856 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.312077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.311869 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.312077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.311972 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.312077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.312009 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "kube-api-access-q475r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.313073 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.312991 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.313169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.313071 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.313766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.313723 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token" pod "cbe5f8f9-9acd-4919-9b79-213f822f8dbf" (UID: "cbe5f8f9-9acd-4919-9b79-213f822f8dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.313766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.313732 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:51.314697 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.314668 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "kube-api-access-lxvqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.315547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.315521 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy" pod "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" (UID: "9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:51.324420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.324399 4285 scope.go:122] "RemoveContainer" containerID="f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875" Nov 02 22:07:51.324747 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:51.324712 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875\": container with ID starting with f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875 not found: ID does not exist" containerID="f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875" Nov 02 22:07:51.324811 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.324749 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875"} err="failed to get container status \"f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875\": rpc error: code = NotFound desc = could not find container \"f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875\": container with ID starting with f6a243fdb394f8c1e859436c9dde2f2219c6f8cfee7f3867d00fda46df448875 not found: ID does not exist" Nov 02 22:07:51.327678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.327652 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf" pod "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" (UID: "a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43"). InnerVolumeSpecName "kube-api-access-l7fsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:51.382050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382019 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382050 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382060 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382069 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382079 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382087 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382095 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382104 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382113 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382134 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382147 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382155 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382163 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvqs\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-kube-api-access-lxvqs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382173 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382182 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382189 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q475r\" (UniqueName: \"kubernetes.io/projected/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-kube-api-access-q475r\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382198 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382205 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382214 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382222 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382230 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382238 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382248 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382257 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382266 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7fsf\" (UniqueName: \"kubernetes.io/projected/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-kube-api-access-l7fsf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382276 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.382295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.382285 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe5f8f9-9acd-4919-9b79-213f822f8dbf-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:51.531306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.531218 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc"] Nov 02 22:07:51.536517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.536491 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-2w6nc"] Nov 02 22:07:51.856996 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.856918 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx"] Nov 02 22:07:51.862841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.862814 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-9mvsx"] Nov 02 22:07:51.965801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.965772 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl"] Nov 02 22:07:51.969792 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:51.969765 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-k7drl"] Nov 02 22:07:52.079606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:52.079574 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p"] Nov 02 22:07:52.083500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:52.083476 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-26z2p"] Nov 02 22:07:52.357468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:52.357433 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:07:52.357637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:52.357485 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:07:52.615489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:52.615411 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:53.176031 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:53.176002 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" path="/var/lib/kubelet/pods/9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a/volumes" Nov 02 22:07:53.176520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:53.176504 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" path="/var/lib/kubelet/pods/a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43/volumes" Nov 02 22:07:53.176941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:53.176929 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" path="/var/lib/kubelet/pods/cbe5f8f9-9acd-4919-9b79-213f822f8dbf/volumes" Nov 02 22:07:53.177606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:53.177588 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" path="/var/lib/kubelet/pods/d7b382c8-0005-4373-8a93-cd6d2ed4212a/volumes" Nov 02 22:07:54.925151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.925110 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 22:07:54.976114 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976085 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976283 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976171 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976283 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976252 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976368 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976372 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:54.976566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976451 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976484 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976510 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976518 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:54.976776 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976573 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976769 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j\") pod \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\" (UID: \"288cffb2-d9ae-4278-ac1a-7a704291bd0a\") " Nov 02 22:07:54.976837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976784 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:54.976948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.976905 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:54.977369 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.977329 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:54.977669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.977637 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:54.977669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.977666 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:54.977801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.977680 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:54.977801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.977694 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:54.977801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.977708 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:54.978938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.978915 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:54.979408 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.979381 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "kube-api-access-gg48j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:54.979516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.979461 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:54.980518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:54.980493 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo" pod "288cffb2-d9ae-4278-ac1a-7a704291bd0a" (UID: "288cffb2-d9ae-4278-ac1a-7a704291bd0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:07:55.079540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.079508 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:55.079540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.079542 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:55.079701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.079558 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg48j\" (UniqueName: \"kubernetes.io/projected/288cffb2-d9ae-4278-ac1a-7a704291bd0a-kube-api-access-gg48j\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:55.079701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.079576 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288cffb2-d9ae-4278-ac1a-7a704291bd0a-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:07:55.276143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.276091 4285 generic.go:356] "Generic (PLEG): container finished" podID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerID="2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2" exitCode=0 Nov 02 22:07:55.276299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.276156 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" event={"ID":"288cffb2-d9ae-4278-ac1a-7a704291bd0a","Type":"ContainerDied","Data":"2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2"} Nov 02 22:07:55.276299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.276173 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" Nov 02 22:07:55.276299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.276200 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8" event={"ID":"288cffb2-d9ae-4278-ac1a-7a704291bd0a","Type":"ContainerDied","Data":"70b0cdce588443598ced2868a5b5620ce3978ca44976a67ece63332af6f7e064"} Nov 02 22:07:55.276299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.276221 4285 scope.go:122] "RemoveContainer" containerID="2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2" Nov 02 22:07:55.288734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.288709 4285 scope.go:122] "RemoveContainer" containerID="2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2" Nov 02 22:07:55.289012 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:07:55.288990 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2\": container with ID starting with 2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2 not found: ID does not exist" containerID="2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2" Nov 02 22:07:55.289082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.289018 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2"} err="failed to get container status \"2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2\": rpc error: code = NotFound desc = could not find container \"2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2\": container with ID starting with 2926c0eb93411f2032ea529c7d83bdf9c4da9c605c79cc498e35c5cce6f50ef2 not found: ID does not exist" Nov 02 22:07:55.441517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.441434 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8"] Nov 02 22:07:55.445114 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:55.445088 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-dnpr8"] Nov 02 22:07:57.178966 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:57.178933 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" path="/var/lib/kubelet/pods/288cffb2-d9ae-4278-ac1a-7a704291bd0a/volumes" Nov 02 22:07:57.237748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:57.237712 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:59.580826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:07:59.580790 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j"] Nov 02 22:08:00.074183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:00.073703 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:00.235046 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:00.235000 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:02.298037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:02.298008 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:04.581181 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:04.581093 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" podUID="b9bc72b6-9d26-4914-a5f8-69364c340da5" containerName="payload-processing" containerID="cri-o://23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1" gracePeriod=25 Nov 02 22:08:04.889475 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:04.889451 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:08:04.999199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:04.999165 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume\") pod \"b9bc72b6-9d26-4914-a5f8-69364c340da5\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " Nov 02 22:08:04.999359 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:04.999294 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz\" (UniqueName: \"kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz\") pod \"b9bc72b6-9d26-4914-a5f8-69364c340da5\" (UID: \"b9bc72b6-9d26-4914-a5f8-69364c340da5\") " Nov 02 22:08:04.999520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:04.999498 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume" pod "b9bc72b6-9d26-4914-a5f8-69364c340da5" (UID: "b9bc72b6-9d26-4914-a5f8-69364c340da5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:04.999812 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:04.999795 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9bc72b6-9d26-4914-a5f8-69364c340da5-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:05.001691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.001664 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz" pod "b9bc72b6-9d26-4914-a5f8-69364c340da5" (UID: "b9bc72b6-9d26-4914-a5f8-69364c340da5"). InnerVolumeSpecName "kube-api-access-28rwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:05.102143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.101563 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rwz\" (UniqueName: \"kubernetes.io/projected/b9bc72b6-9d26-4914-a5f8-69364c340da5-kube-api-access-28rwz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:05.409035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.409004 4285 generic.go:356] "Generic (PLEG): container finished" podID="b9bc72b6-9d26-4914-a5f8-69364c340da5" containerID="23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1" exitCode=0 Nov 02 22:08:05.409233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.409047 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" Nov 02 22:08:05.409233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.409093 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" event={"ID":"b9bc72b6-9d26-4914-a5f8-69364c340da5","Type":"ContainerDied","Data":"23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1"} Nov 02 22:08:05.409233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.409174 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j" event={"ID":"b9bc72b6-9d26-4914-a5f8-69364c340da5","Type":"ContainerDied","Data":"9be5165dd710349e878bfb448a60f91b11040ee276791fef398155a46f09bf40"} Nov 02 22:08:05.409233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.409210 4285 scope.go:122] "RemoveContainer" containerID="23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1" Nov 02 22:08:05.420801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.420783 4285 scope.go:122] "RemoveContainer" containerID="23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1" Nov 02 22:08:05.421068 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:05.421048 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1\": container with ID starting with 23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1 not found: ID does not exist" containerID="23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1" Nov 02 22:08:05.421159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.421075 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1"} err="failed to get container status \"23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1\": rpc error: code = NotFound desc = could not find container \"23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1\": container with ID starting with 23f248e16d06864bdb3d02f9e9c5a34f49924fd7c13717625f36c82620ab89a1 not found: ID does not exist" Nov 02 22:08:05.572848 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.572813 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j"] Nov 02 22:08:05.622517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:05.622486 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-224f53-5967d66c9-67z4j"] Nov 02 22:08:07.175958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:07.175926 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bc72b6-9d26-4914-a5f8-69364c340da5" path="/var/lib/kubelet/pods/b9bc72b6-9d26-4914-a5f8-69364c340da5/volumes" Nov 02 22:08:07.334676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:07.334637 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:09.185591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:09.185551 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk"] Nov 02 22:08:09.451060 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:09.451030 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:09.604460 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:09.604427 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:12.452891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:12.452856 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:13.720895 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:13.720862 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:14.186814 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.186724 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" podUID="ad35d428-e5de-4823-a526-fb35b9f63455" containerName="payload-pre-processing" containerID="cri-o://0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac" gracePeriod=25 Nov 02 22:08:14.463461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.463436 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:08:14.537959 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.537924 4285 generic.go:356] "Generic (PLEG): container finished" podID="ad35d428-e5de-4823-a526-fb35b9f63455" containerID="0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac" exitCode=0 Nov 02 22:08:14.538172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.537970 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" Nov 02 22:08:14.538172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.538013 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" event={"ID":"ad35d428-e5de-4823-a526-fb35b9f63455","Type":"ContainerDied","Data":"0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac"} Nov 02 22:08:14.538172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.538096 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk" event={"ID":"ad35d428-e5de-4823-a526-fb35b9f63455","Type":"ContainerDied","Data":"5a07c73f22a67659e4c4591ff1f1f29f1c1bc1dba27ca38db2ae3f91cb5f739c"} Nov 02 22:08:14.538172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.538120 4285 scope.go:122] "RemoveContainer" containerID="0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac" Nov 02 22:08:14.548309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.548279 4285 scope.go:122] "RemoveContainer" containerID="0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac" Nov 02 22:08:14.548587 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:14.548569 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac\": container with ID starting with 0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac not found: ID does not exist" containerID="0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac" Nov 02 22:08:14.548643 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.548596 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac"} err="failed to get container status \"0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac\": rpc error: code = NotFound desc = could not find container \"0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac\": container with ID starting with 0e5aafdbc56b28325c1ac9be96c48926cc806d0fbe412c5abdbf10e26e9720ac not found: ID does not exist" Nov 02 22:08:14.557191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.557165 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v\" (UniqueName: \"kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v\") pod \"ad35d428-e5de-4823-a526-fb35b9f63455\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " Nov 02 22:08:14.557341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.557328 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume\") pod \"ad35d428-e5de-4823-a526-fb35b9f63455\" (UID: \"ad35d428-e5de-4823-a526-fb35b9f63455\") " Nov 02 22:08:14.557693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.557673 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume" pod "ad35d428-e5de-4823-a526-fb35b9f63455" (UID: "ad35d428-e5de-4823-a526-fb35b9f63455"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:14.557796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.557779 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35d428-e5de-4823-a526-fb35b9f63455-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:14.559236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.559215 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v" pod "ad35d428-e5de-4823-a526-fb35b9f63455" (UID: "ad35d428-e5de-4823-a526-fb35b9f63455"). InnerVolumeSpecName "kube-api-access-nnw4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:14.659601 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.659571 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnw4v\" (UniqueName: \"kubernetes.io/projected/ad35d428-e5de-4823-a526-fb35b9f63455-kube-api-access-nnw4v\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:14.959671 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.959640 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk"] Nov 02 22:08:14.963414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:14.963389 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4fc235-755b47d5f5-jxzwk"] Nov 02 22:08:15.175018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:15.174945 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad35d428-e5de-4823-a526-fb35b9f63455" path="/var/lib/kubelet/pods/ad35d428-e5de-4823-a526-fb35b9f63455/volumes" Nov 02 22:08:16.009391 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:16.009359 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:08:17.211520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:17.211491 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:19.039022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:19.038934 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:19.654031 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:19.653982 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:20.074045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:20.074004 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:21.235326 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.235290 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb"] Nov 02 22:08:21.235920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.235531 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" podUID="3278a470-3837-4abd-baf6-de3a96e2371d" containerName="maas-api" containerID="cri-o://53d0d4b2f68c91e84f03c7d76bf195da565826b92bf7326d7227232280b14d28" gracePeriod=30 Nov 02 22:08:21.363457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.363430 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:21.622021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.621988 4285 generic.go:356] "Generic (PLEG): container finished" podID="3278a470-3837-4abd-baf6-de3a96e2371d" containerID="53d0d4b2f68c91e84f03c7d76bf195da565826b92bf7326d7227232280b14d28" exitCode=0 Nov 02 22:08:21.622213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.622072 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" event={"ID":"3278a470-3837-4abd-baf6-de3a96e2371d","Type":"ContainerDied","Data":"53d0d4b2f68c91e84f03c7d76bf195da565826b92bf7326d7227232280b14d28"} Nov 02 22:08:21.622213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.622171 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" event={"ID":"3278a470-3837-4abd-baf6-de3a96e2371d","Type":"ContainerDied","Data":"43b396870e21d6e8fd806bd83d54a19133a83cf6598d2f2e12d1d06e6cf28c65"} Nov 02 22:08:21.622213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.622204 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43b396870e21d6e8fd806bd83d54a19133a83cf6598d2f2e12d1d06e6cf28c65" Nov 02 22:08:21.643687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.643659 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:08:21.693137 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.693100 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca\") pod \"3278a470-3837-4abd-baf6-de3a96e2371d\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " Nov 02 22:08:21.693294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.693202 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\") pod \"3278a470-3837-4abd-baf6-de3a96e2371d\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " Nov 02 22:08:21.693360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.693340 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs\") pod \"3278a470-3837-4abd-baf6-de3a96e2371d\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " Nov 02 22:08:21.693418 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.693379 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n\" (UniqueName: \"kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n\") pod \"3278a470-3837-4abd-baf6-de3a96e2371d\" (UID: \"3278a470-3837-4abd-baf6-de3a96e2371d\") " Nov 02 22:08:21.693587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.693559 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca" pod "3278a470-3837-4abd-baf6-de3a96e2371d" (UID: "3278a470-3837-4abd-baf6-de3a96e2371d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:21.694039 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.694015 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3278a470-3837-4abd-baf6-de3a96e2371d-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:21.695745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.695476 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls" pod "3278a470-3837-4abd-baf6-de3a96e2371d" (UID: "3278a470-3837-4abd-baf6-de3a96e2371d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:21.695892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.695867 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs" pod "3278a470-3837-4abd-baf6-de3a96e2371d" (UID: "3278a470-3837-4abd-baf6-de3a96e2371d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:21.695957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.695938 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n" pod "3278a470-3837-4abd-baf6-de3a96e2371d" (UID: "3278a470-3837-4abd-baf6-de3a96e2371d"). InnerVolumeSpecName "kube-api-access-ghl2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:21.800364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.800321 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:21.800364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.800363 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3278a470-3837-4abd-baf6-de3a96e2371d-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:21.800618 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:21.800380 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghl2n\" (UniqueName: \"kubernetes.io/projected/3278a470-3837-4abd-baf6-de3a96e2371d-kube-api-access-ghl2n\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:22.629460 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.629432 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb" Nov 02 22:08:22.840235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.840202 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb"] Nov 02 22:08:22.844444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.844408 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4"] Nov 02 22:08:22.845198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845174 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" Nov 02 22:08:22.845198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845197 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845225 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b9bc72b6-9d26-4914-a5f8-69364c340da5" containerName="payload-processing" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845234 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bc72b6-9d26-4914-a5f8-69364c340da5" containerName="payload-processing" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845249 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ad35d428-e5de-4823-a526-fb35b9f63455" containerName="payload-pre-processing" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845254 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad35d428-e5de-4823-a526-fb35b9f63455" containerName="payload-pre-processing" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845268 4285 cpu_manager.go:414] "containerMap: removing container" podUID="6aa9aecd-6b68-459f-82f5-a32118a08a0b" containerName="payload-processing" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845273 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa9aecd-6b68-459f-82f5-a32118a08a0b" containerName="payload-processing" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845285 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3278a470-3837-4abd-baf6-de3a96e2371d" containerName="maas-api" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845290 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3278a470-3837-4abd-baf6-de3a96e2371d" containerName="maas-api" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845303 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845311 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845323 4285 cpu_manager.go:414] "containerMap: removing container" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" Nov 02 22:08:22.845343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845328 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845357 4285 cpu_manager.go:414] "containerMap: removing container" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845364 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845373 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9b4043e2-d29b-44e4-844b-5fc0a2d9201a" containerName="payload-pre-processing" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845380 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4043e2-d29b-44e4-844b-5fc0a2d9201a" containerName="payload-pre-processing" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845391 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845399 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845585 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad35d428-e5de-4823-a526-fb35b9f63455" containerName="payload-pre-processing" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845603 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa9aecd-6b68-459f-82f5-a32118a08a0b" containerName="payload-processing" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845615 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3278a470-3837-4abd-baf6-de3a96e2371d" containerName="maas-api" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845625 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b382c8-0005-4373-8a93-cd6d2ed4212a" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845634 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="288cffb2-d9ae-4278-ac1a-7a704291bd0a" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845648 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe5f8f9-9acd-4919-9b79-213f822f8dbf" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845658 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4043e2-d29b-44e4-844b-5fc0a2d9201a" containerName="payload-pre-processing" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845665 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bb2b3d-8bbe-4709-8f4a-637b7bc2ca43" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845677 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbdd06f-423a-46b9-9ee2-1e07ec32dd0a" containerName="istio-proxy" Nov 02 22:08:22.845940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.845757 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bc72b6-9d26-4914-a5f8-69364c340da5" containerName="payload-processing" Nov 02 22:08:22.850036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.850013 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:22.887997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.887941 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-204000-64cd7c85d8-6b6vb"] Nov 02 22:08:22.915370 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.915343 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2be7cc3-serving-cert\"" Nov 02 22:08:22.936210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.936187 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:22.936308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.936294 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:22.936399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.936385 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:22.936473 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.936458 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4th\" (UniqueName: \"kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:22.949022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.948998 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4"] Nov 02 22:08:22.956103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:22.956078 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.038397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.038360 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.038596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.038438 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4th\" (UniqueName: \"kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.038596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.038537 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.038744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.038639 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.038855 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:23.038832 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-serving-cert: secret "maas-api-e2e-mt-e2be7cc3-serving-cert" not found Nov 02 22:08:23.038924 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:23.038910 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls podName:b0fee743-28c3-48e4-8fe1-326e69e645b3 nodeName:}" failed. No retries permitted until 2026-11-02 22:08:23.538887725 +0000 UTC m=+2042.500438028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls") pod "maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" (UID: "b0fee743-28c3-48e4-8fe1-326e69e645b3") : secret "maas-api-e2e-mt-e2be7cc3-serving-cert" not found Nov 02 22:08:23.039495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.039468 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.042178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.042152 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.105524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.105496 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4th\" (UniqueName: \"kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.175393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.175280 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.195375 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.195337 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3278a470-3837-4abd-baf6-de3a96e2371d" path="/var/lib/kubelet/pods/3278a470-3837-4abd-baf6-de3a96e2371d/volumes" Nov 02 22:08:23.383237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.383195 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.551623 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.551590 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.554094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.554067 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\") pod \"maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.556293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.556266 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.765602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.765572 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:23.901083 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:23.901051 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:24.147115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:24.147090 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4"] Nov 02 22:08:24.148474 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:08:24.148447 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fee743_28c3_48e4_8fe1_326e69e645b3.slice/crio-0239d1f9dbd8418d7ac4b03412ccccc1f9a6156eb555793fc3b8b4cd331fa515 WatchSource:0}: Error finding container 0239d1f9dbd8418d7ac4b03412ccccc1f9a6156eb555793fc3b8b4cd331fa515: Status 404 returned error can't find the container with id 0239d1f9dbd8418d7ac4b03412ccccc1f9a6156eb555793fc3b8b4cd331fa515 Nov 02 22:08:24.150343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:24.150324 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:08:24.644272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:24.644240 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" event={"ID":"b0fee743-28c3-48e4-8fe1-326e69e645b3","Type":"ContainerStarted","Data":"0239d1f9dbd8418d7ac4b03412ccccc1f9a6156eb555793fc3b8b4cd331fa515"} Nov 02 22:08:25.653242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:25.653116 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" event={"ID":"b0fee743-28c3-48e4-8fe1-326e69e645b3","Type":"ContainerStarted","Data":"05df184180fe260b6661ab0b57e478f3cf5c46271db2cae562724ac78e2eab4a"} Nov 02 22:08:25.653242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:25.653223 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:25.769866 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:25.769816 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" podStartSLOduration=2.709572553 podStartE2EDuration="3.769803401s" totalImagesPullingTime="1.060230848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:25.768258669 +0000 UTC m=+2044.729808991" watchObservedRunningTime="2026-11-02 22:08:25.769803401 +0000 UTC m=+2044.731353733" Nov 02 22:08:25.856883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:25.856845 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:26.054741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:26.054704 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:26.268009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:26.267970 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:28.518377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:28.518339 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:29.063300 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.063265 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:29.275012 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.273990 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:29.606304 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.606269 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6"] Nov 02 22:08:29.612327 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.612302 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:29.700156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.699146 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6"] Nov 02 22:08:29.732145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.731501 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17435d\"" Nov 02 22:08:29.732145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.731686 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-17435d-dockercfg-nhn48\"" Nov 02 22:08:29.886564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.886480 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5z8p\" (UniqueName: \"kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p\") pod \"payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:29.886827 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.886809 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:29.988932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.988900 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z8p\" (UniqueName: \"kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p\") pod \"payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:29.989089 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.988962 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:29.989733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:29.989706 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:30.044875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.044839 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z8p\" (UniqueName: \"kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p\") pod \"payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:30.226451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.226410 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:30.603748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.603717 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6"] Nov 02 22:08:30.604951 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:08:30.604917 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016c3683_73b7_41bd_8e88_4f176dbfbdd6.slice/crio-971b1f27b67865a51e312a2b2c75b81d2c40748f30a4fa6e2b3b9f8bc662771f WatchSource:0}: Error finding container 971b1f27b67865a51e312a2b2c75b81d2c40748f30a4fa6e2b3b9f8bc662771f: Status 404 returned error can't find the container with id 971b1f27b67865a51e312a2b2c75b81d2c40748f30a4fa6e2b3b9f8bc662771f Nov 02 22:08:30.702948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.702896 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:30.871113 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.871032 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:30.901184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.901153 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" event={"ID":"016c3683-73b7-41bd-8e88-4f176dbfbdd6","Type":"ContainerStarted","Data":"579e9c51bc0dd50b591e0dd67d2079a49a89bb86887e8ad570ab9d62955d71f3"} Nov 02 22:08:30.901351 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.901212 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" event={"ID":"016c3683-73b7-41bd-8e88-4f176dbfbdd6","Type":"ContainerStarted","Data":"971b1f27b67865a51e312a2b2c75b81d2c40748f30a4fa6e2b3b9f8bc662771f"} Nov 02 22:08:30.901436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:30.901419 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:31.017240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:31.017207 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn"] Nov 02 22:08:31.017467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:31.017440 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" containerID="cri-o://23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc" gracePeriod=30 Nov 02 22:08:31.036545 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:31.036491 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:31.053342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:31.053290 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" podStartSLOduration=2.053273205 podStartE2EDuration="2.053273205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:31.022890065 +0000 UTC m=+2049.984440385" watchObservedRunningTime="2026-11-02 22:08:31.053273205 +0000 UTC m=+2050.014823525" Nov 02 22:08:31.186242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:31.186167 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:31.666154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:31.666094 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:08:33.899659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:33.899563 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:36.310829 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.310806 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:08:36.414522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414483 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.414522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414524 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.414766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414571 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.414766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414602 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.414766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414687 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.414766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414738 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.414766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414760 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.415036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414774 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:36.415036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414872 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.415036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414934 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert\") pod \"edaaab34-eddb-4c62-b119-b51fa28512c9\" (UID: \"edaaab34-eddb-4c62-b119-b51fa28512c9\") " Nov 02 22:08:36.415036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414974 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:36.415036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.414988 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:36.415314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415214 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:36.415553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415517 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:36.415732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415703 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.415732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415734 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.415934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415749 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.415934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415764 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.415934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.415778 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edaaab34-eddb-4c62-b119-b51fa28512c9-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.417083 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.417048 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:08:36.417316 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.417291 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "kube-api-access-hblzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:36.417389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.417345 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:36.418376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.418351 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token" pod "edaaab34-eddb-4c62-b119-b51fa28512c9" (UID: "edaaab34-eddb-4c62-b119-b51fa28512c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:36.518104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.518069 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblzv\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-kube-api-access-hblzv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.518427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.518397 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.518427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.518426 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:36.518617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:36.518439 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edaaab34-eddb-4c62-b119-b51fa28512c9-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:37.000621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.000586 4285 generic.go:356] "Generic (PLEG): container finished" podID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerID="23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc" exitCode=0 Nov 02 22:08:37.000797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.000633 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" Nov 02 22:08:37.000797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.000659 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" event={"ID":"edaaab34-eddb-4c62-b119-b51fa28512c9","Type":"ContainerDied","Data":"23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc"} Nov 02 22:08:37.000797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.000713 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn" event={"ID":"edaaab34-eddb-4c62-b119-b51fa28512c9","Type":"ContainerDied","Data":"12c503567e3e3190d974ce7ccf279e3b35a3e2de4d347d3da6f397b37884483d"} Nov 02 22:08:37.000797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.000733 4285 scope.go:122] "RemoveContainer" containerID="23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc" Nov 02 22:08:37.019430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.019406 4285 scope.go:122] "RemoveContainer" containerID="c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def" Nov 02 22:08:37.030564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.030538 4285 scope.go:122] "RemoveContainer" containerID="23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc" Nov 02 22:08:37.030919 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:37.030891 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc\": container with ID starting with 23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc not found: ID does not exist" containerID="23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc" Nov 02 22:08:37.031010 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.030930 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc"} err="failed to get container status \"23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc\": rpc error: code = NotFound desc = could not find container \"23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc\": container with ID starting with 23e5fffe34f3bef3366d3a90fb4ca3b1214b6e4ae1123200e4a562e47c215dbc not found: ID does not exist" Nov 02 22:08:37.031010 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.030956 4285 scope.go:122] "RemoveContainer" containerID="c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def" Nov 02 22:08:37.031295 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:37.031272 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def\": container with ID starting with c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def not found: ID does not exist" containerID="c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def" Nov 02 22:08:37.031373 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.031304 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def"} err="failed to get container status \"c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def\": rpc error: code = NotFound desc = could not find container \"c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def\": container with ID starting with c6cb657933b56ed468d86e022e7e2d1950b51a2c1375f063e998037354e31def not found: ID does not exist" Nov 02 22:08:37.217325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.217290 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn"] Nov 02 22:08:37.265907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:37.265838 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-204000-openshift-default-7fd9cc7b9f-gtxvn"] Nov 02 22:08:38.591367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.591333 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8"] Nov 02 22:08:38.592245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.592223 4285 cpu_manager.go:414] "containerMap: removing container" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" Nov 02 22:08:38.592245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.592245 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" Nov 02 22:08:38.592419 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.592274 4285 cpu_manager.go:414] "containerMap: removing container" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" Nov 02 22:08:38.592419 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.592281 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" Nov 02 22:08:38.592515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.592493 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" Nov 02 22:08:38.592569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.592527 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" containerName="istio-proxy" Nov 02 22:08:38.597476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.597451 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.653573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.653543 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8"] Nov 02 22:08:38.698176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.698147 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-588b5c-serving-cert\"" Nov 02 22:08:38.751927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.751897 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.752110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.752026 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.752334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.752308 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhbt\" (UniqueName: \"kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.752462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.752444 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.853289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.853205 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:38.853780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.853722 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhbt\" (UniqueName: \"kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.853907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.853859 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.853972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.853904 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.854026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.853997 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.854250 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:38.854231 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-serving-cert: secret "maas-api-e2e-api-a-588b5c-serving-cert" not found Nov 02 22:08:38.854362 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:38.854307 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls podName:e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0 nodeName:}" failed. No retries permitted until 2026-11-02 22:08:39.354287455 +0000 UTC m=+2058.315837767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls") pod "maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" (UID: "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0") : secret "maas-api-e2e-api-a-588b5c-serving-cert" not found Nov 02 22:08:38.854971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.854947 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.857367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.857343 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:38.941210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:38.941171 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhbt\" (UniqueName: \"kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:39.016718 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.016686 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:39.213576 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.213534 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edaaab34-eddb-4c62-b119-b51fa28512c9" path="/var/lib/kubelet/pods/edaaab34-eddb-4c62-b119-b51fa28512c9/volumes" Nov 02 22:08:39.222182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.221108 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:39.366701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.366666 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:39.369149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.369112 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\") pod \"maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:39.510983 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.510906 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:39.894366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:39.894337 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8"] Nov 02 22:08:39.895460 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:08:39.895428 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76aec57_bf6a_4ac3_abbc_bb97aa27f3b0.slice/crio-75cf14051abf0380b22d1432d6ba5666e6530233e858093dd76edc099e709afc WatchSource:0}: Error finding container 75cf14051abf0380b22d1432d6ba5666e6530233e858093dd76edc099e709afc: Status 404 returned error can't find the container with id 75cf14051abf0380b22d1432d6ba5666e6530233e858093dd76edc099e709afc Nov 02 22:08:40.026379 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:40.026347 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" event={"ID":"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0","Type":"ContainerStarted","Data":"75cf14051abf0380b22d1432d6ba5666e6530233e858093dd76edc099e709afc"} Nov 02 22:08:41.042110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:41.042024 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" event={"ID":"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0","Type":"ContainerStarted","Data":"6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c"} Nov 02 22:08:41.042487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:41.042229 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:41.157449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:41.157401 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" podStartSLOduration=2.273709321 podStartE2EDuration="3.157387836s" totalImagesPullingTime="883.678515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:41.154849519 +0000 UTC m=+2060.116399842" watchObservedRunningTime="2026-11-02 22:08:41.157387836 +0000 UTC m=+2060.118938156" Nov 02 22:08:41.911864 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:41.911838 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:08:42.938719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:42.938677 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8"] Nov 02 22:08:43.230676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:43.230641 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:45.604921 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:45.604877 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:46.008578 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:08:46.008539 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:08:47.056234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:47.056201 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:08:47.943288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:47.943227 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" podUID="72063619-3210-4799-a2ea-4f4bdd251d82" containerName="payload-pre-processing" containerID="cri-o://17614e22ac9f829e98694087af15973a9686cfbcfa31f27854d1fcee5ad17dc0" gracePeriod=25 Nov 02 22:08:48.106345 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.106315 4285 generic.go:356] "Generic (PLEG): container finished" podID="72063619-3210-4799-a2ea-4f4bdd251d82" containerID="17614e22ac9f829e98694087af15973a9686cfbcfa31f27854d1fcee5ad17dc0" exitCode=0 Nov 02 22:08:48.106699 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.106357 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" event={"ID":"72063619-3210-4799-a2ea-4f4bdd251d82","Type":"ContainerDied","Data":"17614e22ac9f829e98694087af15973a9686cfbcfa31f27854d1fcee5ad17dc0"} Nov 02 22:08:48.294915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.294879 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:08:48.482777 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.482738 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv\" (UniqueName: \"kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv\") pod \"72063619-3210-4799-a2ea-4f4bdd251d82\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " Nov 02 22:08:48.482976 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.482939 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume\") pod \"72063619-3210-4799-a2ea-4f4bdd251d82\" (UID: \"72063619-3210-4799-a2ea-4f4bdd251d82\") " Nov 02 22:08:48.483358 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.483328 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume" pod "72063619-3210-4799-a2ea-4f4bdd251d82" (UID: "72063619-3210-4799-a2ea-4f4bdd251d82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:48.483919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.483896 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72063619-3210-4799-a2ea-4f4bdd251d82-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:48.485317 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.485288 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv" pod "72063619-3210-4799-a2ea-4f4bdd251d82" (UID: "72063619-3210-4799-a2ea-4f4bdd251d82"). InnerVolumeSpecName "kube-api-access-82frv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:48.585503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:48.585288 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82frv\" (UniqueName: \"kubernetes.io/projected/72063619-3210-4799-a2ea-4f4bdd251d82-kube-api-access-82frv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:08:49.114766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:49.114731 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" event={"ID":"72063619-3210-4799-a2ea-4f4bdd251d82","Type":"ContainerDied","Data":"f18949878acfcbedf90d8ab0331978acd035333c4880fadfc55e6f29ca9eddfc"} Nov 02 22:08:49.114766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:49.114765 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8" Nov 02 22:08:49.115387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:49.114828 4285 scope.go:122] "RemoveContainer" containerID="17614e22ac9f829e98694087af15973a9686cfbcfa31f27854d1fcee5ad17dc0" Nov 02 22:08:49.279204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:49.279168 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8"] Nov 02 22:08:49.328569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:49.328534 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3eb190-9d7c84b5f-v97t8"] Nov 02 22:08:49.744941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:49.744912 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:50.016052 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:50.015967 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:50.174585 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:50.174557 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:50.340318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:50.340236 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:50.609964 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:50.609886 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:51.175147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:51.175096 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72063619-3210-4799-a2ea-4f4bdd251d82" path="/var/lib/kubelet/pods/72063619-3210-4799-a2ea-4f4bdd251d82/volumes" Nov 02 22:08:54.986591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:54.986560 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:55.148960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:55.148909 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:55.629113 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:55.629078 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:56.581827 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:56.581780 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:56.746737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:56.746705 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:56.922289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:56.922170 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:57.619521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:57.619479 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:58.377775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:58.377729 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:58.852343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:08:58.852308 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:00.095101 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:00.095066 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:02.708782 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:02.708747 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:02.867932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:02.867894 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:03.033556 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:03.033462 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:05.295237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:05.295201 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:05.870022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:05.869967 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:06.048486 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:06.048450 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:10.026171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:10.026133 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:11.750509 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.750474 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz"] Nov 02 22:09:11.751181 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.751166 4285 cpu_manager.go:414] "containerMap: removing container" podUID="72063619-3210-4799-a2ea-4f4bdd251d82" containerName="payload-pre-processing" Nov 02 22:09:11.751235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.751182 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72063619-3210-4799-a2ea-4f4bdd251d82" containerName="payload-pre-processing" Nov 02 22:09:11.751314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.751305 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="72063619-3210-4799-a2ea-4f4bdd251d82" containerName="payload-pre-processing" Nov 02 22:09:11.756421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.756396 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.828517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.828487 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.828712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.828684 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.828790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.828730 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.828864 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.828836 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m2l7\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.828949 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.828931 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.828993 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.828977 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.829070 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.829055 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.829177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.829162 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.829255 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.829229 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930249 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930296 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930340 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930398 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930439 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930600 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930550 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930705 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930687 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930789 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930723 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930789 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930777 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m2l7\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930810 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930820 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.930907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.930869 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.931119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.931040 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.931312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.931290 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.932687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.932663 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.933073 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.933050 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.960153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.960112 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz"] Nov 02 22:09:11.987581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.987549 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m2l7\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:11.987988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:11.987967 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:12.071026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:12.070957 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:12.506346 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:12.506317 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz"] Nov 02 22:09:12.506925 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:09:12.506897 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc0981a_7ee2_4396_ab57_5a56d7d16323.slice/crio-09666b120782984a20dfb4ef44c916eabd227652e05cb5984c2e647a6c969aa2 WatchSource:0}: Error finding container 09666b120782984a20dfb4ef44c916eabd227652e05cb5984c2e647a6c969aa2: Status 404 returned error can't find the container with id 09666b120782984a20dfb4ef44c916eabd227652e05cb5984c2e647a6c969aa2 Nov 02 22:09:12.513818 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:12.513786 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:12.513933 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:12.513861 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:12.513933 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:12.513897 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:13.342802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:13.342763 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" event={"ID":"efc0981a-7ee2-4396-ab57-5a56d7d16323","Type":"ContainerStarted","Data":"d48a390937d235b1ad5c1e16d224d16a0c739aba410e93164c7c60d76e42dd43"} Nov 02 22:09:13.343376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:13.342822 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" event={"ID":"efc0981a-7ee2-4396-ab57-5a56d7d16323","Type":"ContainerStarted","Data":"09666b120782984a20dfb4ef44c916eabd227652e05cb5984c2e647a6c969aa2"} Nov 02 22:09:13.511501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:13.511452 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podStartSLOduration=2.511438351 podStartE2EDuration="2.511438351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:13.508886235 +0000 UTC m=+2092.470436554" watchObservedRunningTime="2026-11-02 22:09:13.511438351 +0000 UTC m=+2092.472988701" Nov 02 22:09:14.057528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:14.057494 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:14.072153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:14.072089 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:14.073971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:14.073943 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:09:14.074137 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:14.074000 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 22:09:14.820850 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:14.820742 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:15.072264 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:15.072175 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:09:15.072264 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:15.072234 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 22:09:16.007795 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:09:16.007756 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:09:16.071462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:16.071430 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:09:16.071667 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:16.071485 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 22:09:17.071926 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:17.071888 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:09:17.072347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:17.071948 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 22:09:18.071735 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:18.071703 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:09:18.071937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:18.071760 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 02 22:09:18.985370 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:18.985338 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:19.116682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:19.116648 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:19.118815 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:19.118792 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:19.120364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:19.120345 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:09:20.571390 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:20.571346 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:20.781496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:20.781461 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:20.947461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:20.947428 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:21.449978 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:21.449947 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:23.916711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:23.916676 4285 scope.go:122] "RemoveContainer" containerID="53d0d4b2f68c91e84f03c7d76bf195da565826b92bf7326d7227232280b14d28" Nov 02 22:09:25.852078 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:25.852030 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:26.895566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.895529 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4"] Nov 02 22:09:26.901476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.901448 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945425 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945495 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945538 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945569 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945590 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n2h6\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945731 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945759 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945776 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:26.945899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:26.945877 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.047796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.047754 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.047986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.047807 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.047986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.047853 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.047986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.047885 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n2h6\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048005 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048052 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048082 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048174 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048276 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048486 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048459 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048555 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048524 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048730 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.048844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.048762 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.049057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.049029 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.050492 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.050470 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.051078 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.051058 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.101048 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.101015 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4"] Nov 02 22:09:27.104562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.104514 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.104693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.104609 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n2h6\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6\") pod \"e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.216220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.216188 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:27.624509 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.624423 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:27.649746 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.649719 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4"] Nov 02 22:09:27.650279 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:09:27.650253 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc3b705_119e_4777_a33c_985aa45bd9b8.slice/crio-02ce81bd9b3aedac6c662c2374680aa53fbcec828cc84e425a3c2246239ef62b WatchSource:0}: Error finding container 02ce81bd9b3aedac6c662c2374680aa53fbcec828cc84e425a3c2246239ef62b: Status 404 returned error can't find the container with id 02ce81bd9b3aedac6c662c2374680aa53fbcec828cc84e425a3c2246239ef62b Nov 02 22:09:27.652973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.652940 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:27.653069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.653005 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:27.653069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.653034 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:27.794389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.791684 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:27.982158 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:27.972160 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:28.472296 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:28.472260 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" event={"ID":"9bc3b705-119e-4777-a33c-985aa45bd9b8","Type":"ContainerStarted","Data":"fe3245f72fbc4c718cccd6ba62f759baf91f0f4a316d8466b8cf8df1c96c89a1"} Nov 02 22:09:28.472498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:28.472319 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" event={"ID":"9bc3b705-119e-4777-a33c-985aa45bd9b8","Type":"ContainerStarted","Data":"02ce81bd9b3aedac6c662c2374680aa53fbcec828cc84e425a3c2246239ef62b"} Nov 02 22:09:28.699399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:28.699339 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podStartSLOduration=2.699320876 podStartE2EDuration="2.699320876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:28.64655236 +0000 UTC m=+2107.608102692" watchObservedRunningTime="2026-11-02 22:09:28.699320876 +0000 UTC m=+2107.660871196" Nov 02 22:09:29.229150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:29.219457 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:29.234148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:29.232166 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:09:29.234148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:29.232237 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 22:09:29.838349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:29.838315 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:30.217586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:30.217544 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:09:30.217867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:30.217835 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 22:09:30.514923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:30.514763 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:30.855695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:30.855606 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:31.217505 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:31.217472 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:09:31.217762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:31.217733 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 22:09:31.504410 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:31.504312 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:32.218578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:32.218537 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:09:32.219192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:32.219159 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 22:09:33.219777 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:33.217455 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:09:33.219777 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:33.217509 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 22:09:34.216688 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:34.216652 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:09:34.216860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:34.216714 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 02 22:09:35.220860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:35.220828 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:35.223306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:35.221298 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:35.223306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:35.221807 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:09:36.857596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:36.857535 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:37.036482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:37.036451 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:37.279902 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:37.279872 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:40.004658 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:40.004624 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:40.190312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:40.190276 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.220670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:42.220633 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:46.008602 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:09:46.008563 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:09:46.954370 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:46.954328 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:51.698090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:51.698053 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:56.254770 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:56.254724 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:56.424395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:56.424348 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:56.583385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:56.583292 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:58.339047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:58.339003 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:58.490475 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:09:58.490443 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:01.152194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:01.152105 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:06.242868 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:06.242833 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:11.347147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:11.341221 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:14.397581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:14.397546 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:15.292570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.292530 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8"] Nov 02 22:10:15.292828 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.292800 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" podUID="e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" containerName="maas-api" containerID="cri-o://6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c" gracePeriod=30 Nov 02 22:10:15.431634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.431606 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:15.676573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.676489 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:10:15.838880 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.838832 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\") pod \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " Nov 02 22:10:15.839297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.839260 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt\" (UniqueName: \"kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt\") pod \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " Nov 02 22:10:15.839565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.839547 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca\") pod \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " Nov 02 22:10:15.839943 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.839924 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs\") pod \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\" (UID: \"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0\") " Nov 02 22:10:15.841530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.841497 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca" pod "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" (UID: "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:15.866171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.861400 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls" pod "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" (UID: "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:15.866171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.863547 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt" pod "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" (UID: "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0"). InnerVolumeSpecName "kube-api-access-zvhbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:15.866171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.865257 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs" pod "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" (UID: "e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:15.914472 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.914393 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:15.942268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.942239 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:15.942268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.942266 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:15.942268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.942275 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:15.942491 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:15.942284 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhbt\" (UniqueName: \"kubernetes.io/projected/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0-kube-api-access-zvhbt\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:16.007524 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:10:16.007491 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:10:16.067278 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.067245 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.124218 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.124185 4285 generic.go:356] "Generic (PLEG): container finished" podID="e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" containerID="6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c" exitCode=0 Nov 02 22:10:16.124397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.124230 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" Nov 02 22:10:16.124397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.124265 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" event={"ID":"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0","Type":"ContainerDied","Data":"6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c"} Nov 02 22:10:16.124397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.124336 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8" event={"ID":"e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0","Type":"ContainerDied","Data":"75cf14051abf0380b22d1432d6ba5666e6530233e858093dd76edc099e709afc"} Nov 02 22:10:16.124397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.124366 4285 scope.go:122] "RemoveContainer" containerID="6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c" Nov 02 22:10:16.140354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.140324 4285 scope.go:122] "RemoveContainer" containerID="6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c" Nov 02 22:10:16.140845 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:10:16.140820 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c\": container with ID starting with 6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c not found: ID does not exist" containerID="6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c" Nov 02 22:10:16.140954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.140850 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c"} err="failed to get container status \"6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c\": rpc error: code = NotFound desc = could not find container \"6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c\": container with ID starting with 6644205052c0cf831e0bc8ea3c60d596322a27c18fe1e01c0cbe763428ee7d7c not found: ID does not exist" Nov 02 22:10:16.315260 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.315180 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8"] Nov 02 22:10:16.358170 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.357626 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-588b5c-6cfd85554b-mtqt8"] Nov 02 22:10:16.603581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.603501 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.760147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:16.760033 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:17.174714 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:17.174679 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" path="/var/lib/kubelet/pods/e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0/volumes" Nov 02 22:10:17.213644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:17.213605 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:20.925540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:20.925471 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:21.374564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:21.338565 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:22.005889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:22.005850 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:22.273929 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:22.273847 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:27.490105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:27.490070 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:27.759542 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:27.759456 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:27.924654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:27.924620 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:28.077231 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:28.077142 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:30.083686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:30.083652 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:30.756205 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:30.745601 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:31.190415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:31.190326 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:31.862722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:31.862691 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87"] Nov 02 22:10:31.969578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:31.969543 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt"] Nov 02 22:10:32.268670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:32.268636 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:32.436462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:32.436363 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:32.863281 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:32.863251 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:34.947709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:34.947673 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:35.125668 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:35.125629 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:35.319654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:35.319555 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:36.863005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:36.862959 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" podUID="49a6b374-ffca-44de-ad31-971cff1de3fe" containerName="payload-processing" containerID="cri-o://40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19" gracePeriod=25 Nov 02 22:10:36.970832 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:36.970796 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" podUID="1198c370-1272-41a0-9d26-2d0bcfb3495c" containerName="payload-pre-processing" containerID="cri-o://7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33" gracePeriod=25 Nov 02 22:10:37.164465 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.164437 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:10:37.178634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.178599 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2\" (UniqueName: \"kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2\") pod \"49a6b374-ffca-44de-ad31-971cff1de3fe\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " Nov 02 22:10:37.178796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.178717 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume\") pod \"49a6b374-ffca-44de-ad31-971cff1de3fe\" (UID: \"49a6b374-ffca-44de-ad31-971cff1de3fe\") " Nov 02 22:10:37.179365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.179334 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume" pod "49a6b374-ffca-44de-ad31-971cff1de3fe" (UID: "49a6b374-ffca-44de-ad31-971cff1de3fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:37.181782 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.181760 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2" pod "49a6b374-ffca-44de-ad31-971cff1de3fe" (UID: "49a6b374-ffca-44de-ad31-971cff1de3fe"). InnerVolumeSpecName "kube-api-access-dtvt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:37.253168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.253103 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:10:37.280212 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.280183 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr\" (UniqueName: \"kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr\") pod \"1198c370-1272-41a0-9d26-2d0bcfb3495c\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " Nov 02 22:10:37.280600 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.280558 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume\") pod \"1198c370-1272-41a0-9d26-2d0bcfb3495c\" (UID: \"1198c370-1272-41a0-9d26-2d0bcfb3495c\") " Nov 02 22:10:37.280982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.280953 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume" pod "1198c370-1272-41a0-9d26-2d0bcfb3495c" (UID: "1198c370-1272-41a0-9d26-2d0bcfb3495c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:37.282236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.281361 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1198c370-1272-41a0-9d26-2d0bcfb3495c-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:37.282236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.281391 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtvt2\" (UniqueName: \"kubernetes.io/projected/49a6b374-ffca-44de-ad31-971cff1de3fe-kube-api-access-dtvt2\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:37.282236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.281405 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a6b374-ffca-44de-ad31-971cff1de3fe-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:37.282593 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.282554 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr" pod "1198c370-1272-41a0-9d26-2d0bcfb3495c" (UID: "1198c370-1272-41a0-9d26-2d0bcfb3495c"). InnerVolumeSpecName "kube-api-access-lx5kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:37.331672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.331643 4285 generic.go:356] "Generic (PLEG): container finished" podID="1198c370-1272-41a0-9d26-2d0bcfb3495c" containerID="7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33" exitCode=0 Nov 02 22:10:37.331916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.331674 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" event={"ID":"1198c370-1272-41a0-9d26-2d0bcfb3495c","Type":"ContainerDied","Data":"7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33"} Nov 02 22:10:37.332060 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.331952 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" event={"ID":"1198c370-1272-41a0-9d26-2d0bcfb3495c","Type":"ContainerDied","Data":"01b68d65fe3bf8e9380fb95c6a32527e7b497acbd62bad51c08a46f8e4269e0a"} Nov 02 22:10:37.332060 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.331704 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt" Nov 02 22:10:37.332060 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.331988 4285 scope.go:122] "RemoveContainer" containerID="7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33" Nov 02 22:10:37.333652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.333628 4285 generic.go:356] "Generic (PLEG): container finished" podID="49a6b374-ffca-44de-ad31-971cff1de3fe" containerID="40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19" exitCode=0 Nov 02 22:10:37.333756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.333673 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" Nov 02 22:10:37.333756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.333686 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" event={"ID":"49a6b374-ffca-44de-ad31-971cff1de3fe","Type":"ContainerDied","Data":"40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19"} Nov 02 22:10:37.333756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.333727 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87" event={"ID":"49a6b374-ffca-44de-ad31-971cff1de3fe","Type":"ContainerDied","Data":"c5c02b47fc36b1e2d6b24c691166045fd7b4e37948e4d097fe0c931fbfd0b291"} Nov 02 22:10:37.346157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.346110 4285 scope.go:122] "RemoveContainer" containerID="7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33" Nov 02 22:10:37.349193 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:10:37.347240 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33\": container with ID starting with 7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33 not found: ID does not exist" containerID="7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33" Nov 02 22:10:37.349193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.347282 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33"} err="failed to get container status \"7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33\": rpc error: code = NotFound desc = could not find container \"7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33\": container with ID starting with 7aaca99a3fed06b8395d0876f8d3d2c557663ae926760796eb16516103d17c33 not found: ID does not exist" Nov 02 22:10:37.349193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.347304 4285 scope.go:122] "RemoveContainer" containerID="40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19" Nov 02 22:10:37.360281 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.360262 4285 scope.go:122] "RemoveContainer" containerID="40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19" Nov 02 22:10:37.360574 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:10:37.360554 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19\": container with ID starting with 40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19 not found: ID does not exist" containerID="40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19" Nov 02 22:10:37.360654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.360587 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19"} err="failed to get container status \"40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19\": rpc error: code = NotFound desc = could not find container \"40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19\": container with ID starting with 40c1eaf291e184146d492ee722895318269506a4fd94e0e0d3445e6710c06a19 not found: ID does not exist" Nov 02 22:10:37.383005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.382950 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx5kr\" (UniqueName: \"kubernetes.io/projected/1198c370-1272-41a0-9d26-2d0bcfb3495c-kube-api-access-lx5kr\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:10:37.500479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.500443 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt"] Nov 02 22:10:37.504113 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.504090 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2050f3-7c7cf5c8c9-4g2zt"] Nov 02 22:10:37.611713 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.611683 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87"] Nov 02 22:10:37.615221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.615192 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2050f3-866498b5cc-5jr87"] Nov 02 22:10:37.714621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.714578 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:37.905355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:37.905307 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:38.107581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.107505 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:38.365290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.362552 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:38.527107 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.525529 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm"] Nov 02 22:10:38.527323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527142 4285 cpu_manager.go:414] "containerMap: removing container" podUID="49a6b374-ffca-44de-ad31-971cff1de3fe" containerName="payload-processing" Nov 02 22:10:38.527323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527166 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a6b374-ffca-44de-ad31-971cff1de3fe" containerName="payload-processing" Nov 02 22:10:38.527323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527213 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" containerName="maas-api" Nov 02 22:10:38.527323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527224 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" containerName="maas-api" Nov 02 22:10:38.527323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527287 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1198c370-1272-41a0-9d26-2d0bcfb3495c" containerName="payload-pre-processing" Nov 02 22:10:38.527323 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527296 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1198c370-1272-41a0-9d26-2d0bcfb3495c" containerName="payload-pre-processing" Nov 02 22:10:38.527648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527521 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1198c370-1272-41a0-9d26-2d0bcfb3495c" containerName="payload-pre-processing" Nov 02 22:10:38.527648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527545 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a6b374-ffca-44de-ad31-971cff1de3fe" containerName="payload-processing" Nov 02 22:10:38.527648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.527555 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76aec57-bf6a-4ac3-abbc-bb97aa27f3b0" containerName="maas-api" Nov 02 22:10:38.533768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.533744 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.549517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.549485 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:38.570384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.570354 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm"] Nov 02 22:10:38.588152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.586534 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1789c6\"" Nov 02 22:10:38.588152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.587353 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1789c6-dockercfg-ppd54\"" Nov 02 22:10:38.793527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.793330 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5nh\" (UniqueName: \"kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh\") pod \"payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.794394 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.794367 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.807077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.807051 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r"] Nov 02 22:10:38.825809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.825777 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r"] Nov 02 22:10:38.830513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.830490 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:38.895510 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.895480 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.895664 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.895592 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8hb\" (UniqueName: \"kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb\") pod \"payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:38.895664 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.895613 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:38.895664 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.895636 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5nh\" (UniqueName: \"kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh\") pod \"payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.896186 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.896166 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.951821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.951784 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5nh\" (UniqueName: \"kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh\") pod \"payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:38.996797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.996754 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8hb\" (UniqueName: \"kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb\") pod \"payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:38.996934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.996802 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:38.997574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:38.997554 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:39.056650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.056566 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8hb\" (UniqueName: \"kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb\") pod \"payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:39.154922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.154885 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:39.159236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.159211 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:39.177631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.177604 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1198c370-1272-41a0-9d26-2d0bcfb3495c" path="/var/lib/kubelet/pods/1198c370-1272-41a0-9d26-2d0bcfb3495c/volumes" Nov 02 22:10:39.178103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.178084 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a6b374-ffca-44de-ad31-971cff1de3fe" path="/var/lib/kubelet/pods/49a6b374-ffca-44de-ad31-971cff1de3fe/volumes" Nov 02 22:10:39.419917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.419842 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:39.543740 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:10:39.543696 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f54f65_697b_43af_abbe_f1e530906d7d.slice/crio-27a17fa1f1e6fc2dd846c5957db89c1f80a483b1bbe3864703790802f3ffb066 WatchSource:0}: Error finding container 27a17fa1f1e6fc2dd846c5957db89c1f80a483b1bbe3864703790802f3ffb066: Status 404 returned error can't find the container with id 27a17fa1f1e6fc2dd846c5957db89c1f80a483b1bbe3864703790802f3ffb066 Nov 02 22:10:39.544191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.544119 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm"] Nov 02 22:10:39.567616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.567593 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:39.593099 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:10:39.593060 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc573ebd1_cb9d_4678_89ee_5afccf3b3954.slice/crio-5910eb8fe806f1382183aa446cacabf4ae8abc6c2141143c61f83e7cbeb22e54 WatchSource:0}: Error finding container 5910eb8fe806f1382183aa446cacabf4ae8abc6c2141143c61f83e7cbeb22e54: Status 404 returned error can't find the container with id 5910eb8fe806f1382183aa446cacabf4ae8abc6c2141143c61f83e7cbeb22e54 Nov 02 22:10:39.593252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:39.593176 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r"] Nov 02 22:10:40.394660 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.394622 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" event={"ID":"a4f54f65-697b-43af-abbe-f1e530906d7d","Type":"ContainerStarted","Data":"e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51"} Nov 02 22:10:40.395167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.394690 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:40.395167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.394712 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" event={"ID":"a4f54f65-697b-43af-abbe-f1e530906d7d","Type":"ContainerStarted","Data":"27a17fa1f1e6fc2dd846c5957db89c1f80a483b1bbe3864703790802f3ffb066"} Nov 02 22:10:40.396444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.396417 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" event={"ID":"c573ebd1-cb9d-4678-89ee-5afccf3b3954","Type":"ContainerStarted","Data":"8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7"} Nov 02 22:10:40.396578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.396469 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" event={"ID":"c573ebd1-cb9d-4678-89ee-5afccf3b3954","Type":"ContainerStarted","Data":"5910eb8fe806f1382183aa446cacabf4ae8abc6c2141143c61f83e7cbeb22e54"} Nov 02 22:10:40.396578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.396497 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:40.624014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.623957 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" podStartSLOduration=2.623943335 podStartE2EDuration="2.623943335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:40.509262094 +0000 UTC m=+2179.470812415" watchObservedRunningTime="2026-11-02 22:10:40.623943335 +0000 UTC m=+2179.585493654" Nov 02 22:10:40.625615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:40.625573 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" podStartSLOduration=2.625561652 podStartE2EDuration="2.625561652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:40.621895512 +0000 UTC m=+2179.583445833" watchObservedRunningTime="2026-11-02 22:10:40.625561652 +0000 UTC m=+2179.587111992" Nov 02 22:10:42.994548 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:42.994507 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:46.009245 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:10:46.009209 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:10:47.651799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:47.651761 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:47.817727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:47.817682 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:51.408025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:51.407989 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:10:51.408508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:51.408073 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:10:52.310094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:52.310042 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:56.370903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:56.370856 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:57.738682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:57.738644 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:58.126277 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:58.126179 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:58.285121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:10:58.285084 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:03.200438 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:03.200383 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:03.425882 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:03.425849 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:03.572642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:03.572533 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:03.715746 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:03.715713 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr"] Nov 02 22:11:03.716003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:03.715960 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" containerID="cri-o://c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536" gracePeriod=30 Nov 02 22:11:03.742087 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:03.742057 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:08.264879 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:08.264842 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:08.998058 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:08.997992 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:11:09.172594 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.172563 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.172797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.172624 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.172797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.172650 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.172797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.172756 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.172959 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.172801 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.172959 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.172879 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.173060 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.173035 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.173368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.173168 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.173368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.173215 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9\") pod \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\" (UID: \"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10\") " Nov 02 22:11:09.173955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.173665 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:09.174502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.174203 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.174502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.174448 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:09.174867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.174676 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:09.175135 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.174916 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:09.177568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.177533 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:09.177954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.177929 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:09.178500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.178475 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "kube-api-access-kr6v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:09.179340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.179315 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:11:09.186553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.186523 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data" pod "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" (UID: "1a60fab5-e08f-49b8-bf6f-0ed7c1260c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275464 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275500 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275513 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275525 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275537 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275548 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6v9\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-kube-api-access-kr6v9\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.275563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275562 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.276171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.275574 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:09.640466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.640383 4285 generic.go:356] "Generic (PLEG): container finished" podID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerID="c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536" exitCode=0 Nov 02 22:11:09.640466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.640438 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" Nov 02 22:11:09.640650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.640470 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" event={"ID":"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10","Type":"ContainerDied","Data":"c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536"} Nov 02 22:11:09.640650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.640530 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr" event={"ID":"1a60fab5-e08f-49b8-bf6f-0ed7c1260c10","Type":"ContainerDied","Data":"0fdf3c09d3b2a0854bf7e86628a2614f5a60b16907ca3b93077c8ec1b29110de"} Nov 02 22:11:09.640650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.640553 4285 scope.go:122] "RemoveContainer" containerID="c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536" Nov 02 22:11:09.651435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.651415 4285 scope.go:122] "RemoveContainer" containerID="4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4" Nov 02 22:11:09.662164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.662137 4285 scope.go:122] "RemoveContainer" containerID="c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536" Nov 02 22:11:09.662509 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:09.662487 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536\": container with ID starting with c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536 not found: ID does not exist" containerID="c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536" Nov 02 22:11:09.662616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.662522 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536"} err="failed to get container status \"c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536\": rpc error: code = NotFound desc = could not find container \"c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536\": container with ID starting with c9f92d84affe7cb7b83ae0bc0e9956043b424a6f33cc75b3e8c45005d0424536 not found: ID does not exist" Nov 02 22:11:09.662616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.662548 4285 scope.go:122] "RemoveContainer" containerID="4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4" Nov 02 22:11:09.662849 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:09.662832 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4\": container with ID starting with 4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4 not found: ID does not exist" containerID="4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4" Nov 02 22:11:09.662917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.662857 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4"} err="failed to get container status \"4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4\": rpc error: code = NotFound desc = could not find container \"4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4\": container with ID starting with 4b2696b567d8a141ff010eca15915eccce872cd15251a47a96f274ac8aa4f9a4 not found: ID does not exist" Nov 02 22:11:09.804848 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.804813 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr"] Nov 02 22:11:09.855407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:09.855375 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4fc235-openshift-default-84fdb6595-wnjhr"] Nov 02 22:11:11.175466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:11.175436 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" path="/var/lib/kubelet/pods/1a60fab5-e08f-49b8-bf6f-0ed7c1260c10/volumes" Nov 02 22:11:12.936041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.935999 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw"] Nov 02 22:11:12.936745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.936730 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" Nov 02 22:11:12.936788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.936745 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" Nov 02 22:11:12.936788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.936773 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" Nov 02 22:11:12.936788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.936779 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" Nov 02 22:11:12.936899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.936886 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" Nov 02 22:11:12.941974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.941954 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:12.996402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:12.996364 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw"] Nov 02 22:11:13.027113 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.027070 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.027329 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.027286 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkh47\" (UniqueName: \"kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.027473 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.027453 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.027533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.027484 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.041740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.041711 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-027bdf-serving-cert\"" Nov 02 22:11:13.129092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.129048 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkh47\" (UniqueName: \"kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.129291 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.129247 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.129291 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.129278 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.129382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.129368 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.129518 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:13.129501 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-serving-cert: secret "maas-api-e2e-inf-a-027bdf-serving-cert" not found Nov 02 22:11:13.129599 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:13.129588 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls podName:3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd nodeName:}" failed. No retries permitted until 2026-11-02 22:11:13.629566263 +0000 UTC m=+2212.591116574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls") pod "maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" (UID: "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd") : secret "maas-api-e2e-inf-a-027bdf-serving-cert" not found Nov 02 22:11:13.129891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.129867 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.131883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.131856 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.154891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.154858 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:13.227237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.226966 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkh47\" (UniqueName: \"kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.309294 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.309261 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:13.576243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.576160 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:13.641804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.641768 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.644301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.644274 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\") pod \"maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:13.855513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:13.855437 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:14.230732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:14.230702 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw"] Nov 02 22:11:14.231382 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:11:14.231355 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbaaeca_f6a0_49eb_a5e8_8a1d8fd471fd.slice/crio-daf448220f361533c6be34932d265b3c2fbd334ec1ccd51832287d9308f512e3 WatchSource:0}: Error finding container daf448220f361533c6be34932d265b3c2fbd334ec1ccd51832287d9308f512e3: Status 404 returned error can't find the container with id daf448220f361533c6be34932d265b3c2fbd334ec1ccd51832287d9308f512e3 Nov 02 22:11:14.678468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:14.678384 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" event={"ID":"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd","Type":"ContainerStarted","Data":"daf448220f361533c6be34932d265b3c2fbd334ec1ccd51832287d9308f512e3"} Nov 02 22:11:14.716875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:14.716843 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:15.697748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:15.697694 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" event={"ID":"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd","Type":"ContainerStarted","Data":"f9ca3af78e4d149e96f7fcb8131fca22211db4a48345d1871eb9823b5ab50ec2"} Nov 02 22:11:15.811577 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:15.811512 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" podStartSLOduration=2.98492617 podStartE2EDuration="3.811496767s" totalImagesPullingTime="826.570597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:15.810470043 +0000 UTC m=+2214.772020363" watchObservedRunningTime="2026-11-02 22:11:15.811496767 +0000 UTC m=+2214.773047097" Nov 02 22:11:16.088780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:16.088686 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz"] Nov 02 22:11:16.111827 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:16.111787 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:11:16.449731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:16.449705 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:16.614228 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:16.601977 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:16.705808 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:16.705733 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:18.148215 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:18.148176 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:21.089777 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.089736 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" podUID="e8e87085-4bd5-42cf-8599-1a916c6bb3be" containerName="payload-processing" containerID="cri-o://44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157" gracePeriod=25 Nov 02 22:11:21.384386 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.384363 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:11:21.506348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.506319 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf\" (UniqueName: \"kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf\") pod \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " Nov 02 22:11:21.506524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.506403 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume\") pod \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\" (UID: \"e8e87085-4bd5-42cf-8599-1a916c6bb3be\") " Nov 02 22:11:21.506822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.506796 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume" pod "e8e87085-4bd5-42cf-8599-1a916c6bb3be" (UID: "e8e87085-4bd5-42cf-8599-1a916c6bb3be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:21.506987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.506970 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e87085-4bd5-42cf-8599-1a916c6bb3be-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:21.508864 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.508839 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf" pod "e8e87085-4bd5-42cf-8599-1a916c6bb3be" (UID: "e8e87085-4bd5-42cf-8599-1a916c6bb3be"). InnerVolumeSpecName "kube-api-access-d6xhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:21.608570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.608506 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6xhf\" (UniqueName: \"kubernetes.io/projected/e8e87085-4bd5-42cf-8599-1a916c6bb3be-kube-api-access-d6xhf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:21.744754 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.744718 4285 generic.go:356] "Generic (PLEG): container finished" podID="e8e87085-4bd5-42cf-8599-1a916c6bb3be" containerID="44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157" exitCode=0 Nov 02 22:11:21.744923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.744766 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" Nov 02 22:11:21.744923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.744797 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" event={"ID":"e8e87085-4bd5-42cf-8599-1a916c6bb3be","Type":"ContainerDied","Data":"44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157"} Nov 02 22:11:21.744923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.744852 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz" event={"ID":"e8e87085-4bd5-42cf-8599-1a916c6bb3be","Type":"ContainerDied","Data":"d3c4bbb3faa40f19cf835861f506d3f281ec607a62843e9e7b0fc5e38b750fe6"} Nov 02 22:11:21.744923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.744874 4285 scope.go:122] "RemoveContainer" containerID="44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157" Nov 02 22:11:21.755732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.755713 4285 scope.go:122] "RemoveContainer" containerID="44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157" Nov 02 22:11:21.755993 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:21.755974 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157\": container with ID starting with 44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157 not found: ID does not exist" containerID="44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157" Nov 02 22:11:21.756056 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.756001 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157"} err="failed to get container status \"44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157\": rpc error: code = NotFound desc = could not find container \"44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157\": container with ID starting with 44ccbb1f19daccbf3655a7c3dc90762300930c04f6a4dc8f93aa4cabcacfd157 not found: ID does not exist" Nov 02 22:11:21.907992 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.907927 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz"] Nov 02 22:11:21.959427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:21.959398 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-587629-589679fb55-9vhrz"] Nov 02 22:11:22.718961 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:22.718931 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:11:22.803824 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:22.803793 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:23.176026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:23.175940 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e87085-4bd5-42cf-8599-1a916c6bb3be" path="/var/lib/kubelet/pods/e8e87085-4bd5-42cf-8599-1a916c6bb3be/volumes" Nov 02 22:11:27.676918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:27.652744 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:32.601745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:32.601716 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:37.343002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:37.342960 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:39.559453 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:39.559417 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:39.729470 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:39.729427 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:39.816015 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:39.815921 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6"] Nov 02 22:11:40.233657 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:40.233620 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:40.414961 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:40.414923 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:41.389421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:41.389367 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4"] Nov 02 22:11:41.394143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:41.390638 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" podUID="b0fee743-28c3-48e4-8fe1-326e69e645b3" containerName="maas-api" containerID="cri-o://05df184180fe260b6661ab0b57e478f3cf5c46271db2cae562724ac78e2eab4a" gracePeriod=30 Nov 02 22:11:41.541987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:41.541930 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:41.972905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:41.972644 4285 generic.go:356] "Generic (PLEG): container finished" podID="b0fee743-28c3-48e4-8fe1-326e69e645b3" containerID="05df184180fe260b6661ab0b57e478f3cf5c46271db2cae562724ac78e2eab4a" exitCode=0 Nov 02 22:11:41.972905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:41.972881 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" event={"ID":"b0fee743-28c3-48e4-8fe1-326e69e645b3","Type":"ContainerDied","Data":"05df184180fe260b6661ab0b57e478f3cf5c46271db2cae562724ac78e2eab4a"} Nov 02 22:11:42.032920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.032896 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:11:42.094439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.094413 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\") pod \"b0fee743-28c3-48e4-8fe1-326e69e645b3\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " Nov 02 22:11:42.094606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.094471 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca\") pod \"b0fee743-28c3-48e4-8fe1-326e69e645b3\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " Nov 02 22:11:42.094606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.094529 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th\" (UniqueName: \"kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th\") pod \"b0fee743-28c3-48e4-8fe1-326e69e645b3\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " Nov 02 22:11:42.094730 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.094699 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs\") pod \"b0fee743-28c3-48e4-8fe1-326e69e645b3\" (UID: \"b0fee743-28c3-48e4-8fe1-326e69e645b3\") " Nov 02 22:11:42.094892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.094869 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca" pod "b0fee743-28c3-48e4-8fe1-326e69e645b3" (UID: "b0fee743-28c3-48e4-8fe1-326e69e645b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:42.095343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.095320 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fee743-28c3-48e4-8fe1-326e69e645b3-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:42.096902 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.096841 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls" pod "b0fee743-28c3-48e4-8fe1-326e69e645b3" (UID: "b0fee743-28c3-48e4-8fe1-326e69e645b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:42.097226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.097200 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs" pod "b0fee743-28c3-48e4-8fe1-326e69e645b3" (UID: "b0fee743-28c3-48e4-8fe1-326e69e645b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:42.098343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.098316 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th" pod "b0fee743-28c3-48e4-8fe1-326e69e645b3" (UID: "b0fee743-28c3-48e4-8fe1-326e69e645b3"). InnerVolumeSpecName "kube-api-access-gz4th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:42.196780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.196746 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4th\" (UniqueName: \"kubernetes.io/projected/b0fee743-28c3-48e4-8fe1-326e69e645b3-kube-api-access-gz4th\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:42.196780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.196779 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:42.196969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.196794 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fee743-28c3-48e4-8fe1-326e69e645b3-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:42.981498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.981465 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" Nov 02 22:11:42.981498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.981482 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" event={"ID":"b0fee743-28c3-48e4-8fe1-326e69e645b3","Type":"ContainerDied","Data":"0239d1f9dbd8418d7ac4b03412ccccc1f9a6156eb555793fc3b8b4cd331fa515"} Nov 02 22:11:42.982036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.981562 4285 scope.go:122] "RemoveContainer" containerID="05df184180fe260b6661ab0b57e478f3cf5c46271db2cae562724ac78e2eab4a" Nov 02 22:11:42.994237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:42.994214 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:43.144950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:43.144916 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4"] Nov 02 22:11:43.194831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:43.194796 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4"] Nov 02 22:11:43.231027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:43.231002 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:43.390861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:43.390773 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:43.943603 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:43.943562 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:44.778110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:44.778063 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2be7cc3-65f849d6c6-tqbz4" podUID="b0fee743-28c3-48e4-8fe1-326e69e645b3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:44.817319 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:44.817267 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" podUID="016c3683-73b7-41bd-8e88-4f176dbfbdd6" containerName="payload-processing" containerID="cri-o://579e9c51bc0dd50b591e0dd67d2079a49a89bb86887e8ad570ab9d62955d71f3" gracePeriod=25 Nov 02 22:11:45.020867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.020452 4285 generic.go:356] "Generic (PLEG): container finished" podID="016c3683-73b7-41bd-8e88-4f176dbfbdd6" containerID="579e9c51bc0dd50b591e0dd67d2079a49a89bb86887e8ad570ab9d62955d71f3" exitCode=0 Nov 02 22:11:45.020867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.020512 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" event={"ID":"016c3683-73b7-41bd-8e88-4f176dbfbdd6","Type":"ContainerDied","Data":"579e9c51bc0dd50b591e0dd67d2079a49a89bb86887e8ad570ab9d62955d71f3"} Nov 02 22:11:45.118777 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.118747 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:11:45.150562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.150535 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume\") pod \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " Nov 02 22:11:45.150727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.150680 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p\" (UniqueName: \"kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p\") pod \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\" (UID: \"016c3683-73b7-41bd-8e88-4f176dbfbdd6\") " Nov 02 22:11:45.150974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.150948 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume" pod "016c3683-73b7-41bd-8e88-4f176dbfbdd6" (UID: "016c3683-73b7-41bd-8e88-4f176dbfbdd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:45.151439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.151418 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/016c3683-73b7-41bd-8e88-4f176dbfbdd6-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:45.152876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.152855 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p" pod "016c3683-73b7-41bd-8e88-4f176dbfbdd6" (UID: "016c3683-73b7-41bd-8e88-4f176dbfbdd6"). InnerVolumeSpecName "kube-api-access-v5z8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:45.174513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.174484 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fee743-28c3-48e4-8fe1-326e69e645b3" path="/var/lib/kubelet/pods/b0fee743-28c3-48e4-8fe1-326e69e645b3/volumes" Nov 02 22:11:45.253571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:45.253517 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5z8p\" (UniqueName: \"kubernetes.io/projected/016c3683-73b7-41bd-8e88-4f176dbfbdd6-kube-api-access-v5z8p\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:11:46.009275 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:11:46.009092 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:11:46.028963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:46.028937 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" Nov 02 22:11:46.029307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:46.028997 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6" event={"ID":"016c3683-73b7-41bd-8e88-4f176dbfbdd6","Type":"ContainerDied","Data":"971b1f27b67865a51e312a2b2c75b81d2c40748f30a4fa6e2b3b9f8bc662771f"} Nov 02 22:11:46.029445 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:46.029342 4285 scope.go:122] "RemoveContainer" containerID="579e9c51bc0dd50b591e0dd67d2079a49a89bb86887e8ad570ab9d62955d71f3" Nov 02 22:11:46.191073 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:46.191041 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6"] Nov 02 22:11:46.196136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:46.196104 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17435d-8759494d9-9tqs6"] Nov 02 22:11:47.175257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:47.175224 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016c3683-73b7-41bd-8e88-4f176dbfbdd6" path="/var/lib/kubelet/pods/016c3683-73b7-41bd-8e88-4f176dbfbdd6/volumes" Nov 02 22:11:48.397432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:48.397399 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:48.850710 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:48.850669 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv"] Nov 02 22:11:48.942701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:48.942667 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:53.587495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:53.587458 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:58.206632 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:58.206600 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:58.672794 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:11:58.672715 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:02.273648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:02.273608 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:03.796553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:03.796523 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:03.873355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:03.873314 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" podUID="af54beaf-c72f-40c6-adaa-a122e0d0301b" containerName="main" containerID="cri-o://a166fb59034da455625e1c7b28d2ec570178db14b6168f8de138506b2e40ad62" gracePeriod=45 Nov 02 22:12:04.003642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.003614 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:04.199630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.199589 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:04.207576 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.207546 4285 generic.go:356] "Generic (PLEG): container finished" podID="af54beaf-c72f-40c6-adaa-a122e0d0301b" containerID="a166fb59034da455625e1c7b28d2ec570178db14b6168f8de138506b2e40ad62" exitCode=0 Nov 02 22:12:04.207699 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.207596 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" event={"ID":"af54beaf-c72f-40c6-adaa-a122e0d0301b","Type":"ContainerDied","Data":"a166fb59034da455625e1c7b28d2ec570178db14b6168f8de138506b2e40ad62"} Nov 02 22:12:04.226907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.226883 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:12:04.282424 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282391 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv\" (UniqueName: \"kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv\") pod \"af54beaf-c72f-40c6-adaa-a122e0d0301b\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " Nov 02 22:12:04.282596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282477 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir\") pod \"af54beaf-c72f-40c6-adaa-a122e0d0301b\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " Nov 02 22:12:04.282643 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282628 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs\" (UniqueName: \"kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs\") pod \"af54beaf-c72f-40c6-adaa-a122e0d0301b\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " Nov 02 22:12:04.282705 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282653 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache\") pod \"af54beaf-c72f-40c6-adaa-a122e0d0301b\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " Nov 02 22:12:04.282763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282711 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm\") pod \"af54beaf-c72f-40c6-adaa-a122e0d0301b\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " Nov 02 22:12:04.282763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282723 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir" pod "af54beaf-c72f-40c6-adaa-a122e0d0301b" (UID: "af54beaf-c72f-40c6-adaa-a122e0d0301b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:04.282763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282756 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home\") pod \"af54beaf-c72f-40c6-adaa-a122e0d0301b\" (UID: \"af54beaf-c72f-40c6-adaa-a122e0d0301b\") " Nov 02 22:12:04.283023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.282979 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache" pod "af54beaf-c72f-40c6-adaa-a122e0d0301b" (UID: "af54beaf-c72f-40c6-adaa-a122e0d0301b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:04.283140 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.283052 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home" pod "af54beaf-c72f-40c6-adaa-a122e0d0301b" (UID: "af54beaf-c72f-40c6-adaa-a122e0d0301b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:04.283332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.283312 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.283416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.283336 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.283416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.283351 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.284750 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.284721 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs" pod "af54beaf-c72f-40c6-adaa-a122e0d0301b" (UID: "af54beaf-c72f-40c6-adaa-a122e0d0301b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:04.284860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.284795 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm" pod "af54beaf-c72f-40c6-adaa-a122e0d0301b" (UID: "af54beaf-c72f-40c6-adaa-a122e0d0301b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:04.284860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.284831 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv" pod "af54beaf-c72f-40c6-adaa-a122e0d0301b" (UID: "af54beaf-c72f-40c6-adaa-a122e0d0301b"). InnerVolumeSpecName "kube-api-access-dzbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:04.384877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.384784 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzbjv\" (UniqueName: \"kubernetes.io/projected/af54beaf-c72f-40c6-adaa-a122e0d0301b-kube-api-access-dzbjv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.384877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.384813 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af54beaf-c72f-40c6-adaa-a122e0d0301b-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.384877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.384823 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af54beaf-c72f-40c6-adaa-a122e0d0301b-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.660971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:04.660844 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:05.062097 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:05.062052 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" podUID="af54beaf-c72f-40c6-adaa-a122e0d0301b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:05.216464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:05.216432 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" Nov 02 22:12:05.216642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:05.216439 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv" event={"ID":"af54beaf-c72f-40c6-adaa-a122e0d0301b","Type":"ContainerDied","Data":"e3cd222fa0bb2db9814eedbad6ba06903d6e99bc0889b259df64713207540b68"} Nov 02 22:12:05.216642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:05.216549 4285 scope.go:122] "RemoveContainer" containerID="a166fb59034da455625e1c7b28d2ec570178db14b6168f8de138506b2e40ad62" Nov 02 22:12:05.382033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:05.381940 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv"] Nov 02 22:12:05.431386 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:05.431354 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e2be7cc3/e2e-lifecycle-model-e2be7cc3-kserve-79c7f4c9c7-h5ctv"] Nov 02 22:12:07.174807 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:07.174772 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af54beaf-c72f-40c6-adaa-a122e0d0301b" path="/var/lib/kubelet/pods/af54beaf-c72f-40c6-adaa-a122e0d0301b/volumes" Nov 02 22:12:08.864762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:08.864727 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:09.015772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:09.015732 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:09.156591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:09.156486 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:10.464967 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:10.464936 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:10.628912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:10.628869 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:10.834761 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:10.834660 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:14.199057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:14.199013 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:15.048622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:15.048559 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:16.006696 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:16.006655 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:12:18.783341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:18.783305 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:18.997729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:18.997628 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:19.141188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:19.141054 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:23.924619 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:23.924579 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:24.406315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:24.406276 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4"] Nov 02 22:12:24.673041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:24.672951 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:24.826264 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:24.826230 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:25.246950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:25.246917 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:26.066558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:26.066516 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:28.705882 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:28.705844 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:29.407322 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.407277 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" podUID="b2c3a9a9-5ffd-4638-9986-36ff5876e38f" containerName="payload-pre-processing" containerID="cri-o://8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35" gracePeriod=25 Nov 02 22:12:29.704827 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.704798 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:12:29.807926 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.807894 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume\") pod \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " Nov 02 22:12:29.808346 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.808170 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29\" (UniqueName: \"kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29\") pod \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\" (UID: \"b2c3a9a9-5ffd-4638-9986-36ff5876e38f\") " Nov 02 22:12:29.808346 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.808280 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume" pod "b2c3a9a9-5ffd-4638-9986-36ff5876e38f" (UID: "b2c3a9a9-5ffd-4638-9986-36ff5876e38f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:29.808717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.808698 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:29.810188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.810167 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29" pod "b2c3a9a9-5ffd-4638-9986-36ff5876e38f" (UID: "b2c3a9a9-5ffd-4638-9986-36ff5876e38f"). InnerVolumeSpecName "kube-api-access-wzz29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:29.910434 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:29.910397 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzz29\" (UniqueName: \"kubernetes.io/projected/b2c3a9a9-5ffd-4638-9986-36ff5876e38f-kube-api-access-wzz29\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:30.422731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.422692 4285 generic.go:356] "Generic (PLEG): container finished" podID="b2c3a9a9-5ffd-4638-9986-36ff5876e38f" containerID="8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35" exitCode=0 Nov 02 22:12:30.422919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.422747 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" Nov 02 22:12:30.422919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.422773 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" event={"ID":"b2c3a9a9-5ffd-4638-9986-36ff5876e38f","Type":"ContainerDied","Data":"8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35"} Nov 02 22:12:30.422919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.422845 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4" event={"ID":"b2c3a9a9-5ffd-4638-9986-36ff5876e38f","Type":"ContainerDied","Data":"924727cab285b1a6e6689b165398f7f0fb762fb3aefd3136128e588f982cf7e6"} Nov 02 22:12:30.422919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.422877 4285 scope.go:122] "RemoveContainer" containerID="8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35" Nov 02 22:12:30.434800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.434782 4285 scope.go:122] "RemoveContainer" containerID="8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35" Nov 02 22:12:30.435111 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:30.435087 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35\": container with ID starting with 8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35 not found: ID does not exist" containerID="8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35" Nov 02 22:12:30.435238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.435133 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35"} err="failed to get container status \"8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35\": rpc error: code = NotFound desc = could not find container \"8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35\": container with ID starting with 8983a9a3cbad4044bc0d48a973247efdb823339524f616062049bf60bc1c2d35 not found: ID does not exist" Nov 02 22:12:30.587344 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.587268 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4"] Nov 02 22:12:30.638825 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:30.638785 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d1c62e-689fc956d7-94dm4"] Nov 02 22:12:31.175512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:31.175471 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c3a9a9-5ffd-4638-9986-36ff5876e38f" path="/var/lib/kubelet/pods/b2c3a9a9-5ffd-4638-9986-36ff5876e38f/volumes" Nov 02 22:12:31.641237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:31.641206 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:31.882466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:31.882414 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:31.997823 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:31.997792 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6"] Nov 02 22:12:31.998053 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:31.998030 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" containerID="cri-o://2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff" gracePeriod=30 Nov 02 22:12:32.012692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.012658 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz"] Nov 02 22:12:32.012855 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.012834 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" containerID="cri-o://d48a390937d235b1ad5c1e16d224d16a0c739aba410e93164c7c60d76e42dd43" gracePeriod=30 Nov 02 22:12:32.033467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.033438 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4"] Nov 02 22:12:32.033630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.033609 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" containerID="cri-o://fe3245f72fbc4c718cccd6ba62f759baf91f0f4a316d8466b8cf8df1c96c89a1" gracePeriod=30 Nov 02 22:12:32.047455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.047426 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:32.698334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.698295 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn"] Nov 02 22:12:32.698802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.698583 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" podUID="65d8a917-ecae-4593-b818-7a296d45f8fd" containerName="maas-api" containerID="cri-o://4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e" gracePeriod=30 Nov 02 22:12:32.824057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:32.824026 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:33.050952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.050928 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:12:33.164190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.164145 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca\") pod \"65d8a917-ecae-4593-b818-7a296d45f8fd\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " Nov 02 22:12:33.164506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.164487 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs\") pod \"65d8a917-ecae-4593-b818-7a296d45f8fd\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " Nov 02 22:12:33.164647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.164634 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\") pod \"65d8a917-ecae-4593-b818-7a296d45f8fd\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " Nov 02 22:12:33.164849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.164837 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm\" (UniqueName: \"kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm\") pod \"65d8a917-ecae-4593-b818-7a296d45f8fd\" (UID: \"65d8a917-ecae-4593-b818-7a296d45f8fd\") " Nov 02 22:12:33.166590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.166501 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca" pod "65d8a917-ecae-4593-b818-7a296d45f8fd" (UID: "65d8a917-ecae-4593-b818-7a296d45f8fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:33.175183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.174758 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls" pod "65d8a917-ecae-4593-b818-7a296d45f8fd" (UID: "65d8a917-ecae-4593-b818-7a296d45f8fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:33.193926 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.179680 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm" pod "65d8a917-ecae-4593-b818-7a296d45f8fd" (UID: "65d8a917-ecae-4593-b818-7a296d45f8fd"). InnerVolumeSpecName "kube-api-access-tklbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:33.208367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.208252 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs" pod "65d8a917-ecae-4593-b818-7a296d45f8fd" (UID: "65d8a917-ecae-4593-b818-7a296d45f8fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:33.267082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.266994 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d8a917-ecae-4593-b818-7a296d45f8fd-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:33.267082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.267023 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:33.267082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.267036 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d8a917-ecae-4593-b818-7a296d45f8fd-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:33.267082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.267050 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklbm\" (UniqueName: \"kubernetes.io/projected/65d8a917-ecae-4593-b818-7a296d45f8fd-kube-api-access-tklbm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:33.449373 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.449327 4285 generic.go:356] "Generic (PLEG): container finished" podID="65d8a917-ecae-4593-b818-7a296d45f8fd" containerID="4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e" exitCode=0 Nov 02 22:12:33.449584 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.449435 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" Nov 02 22:12:33.449584 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.449438 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" event={"ID":"65d8a917-ecae-4593-b818-7a296d45f8fd","Type":"ContainerDied","Data":"4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e"} Nov 02 22:12:33.449584 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.449576 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn" event={"ID":"65d8a917-ecae-4593-b818-7a296d45f8fd","Type":"ContainerDied","Data":"b939e0efb601f3abdb504ce4445b7955802f7e9806fb8934ff85ded440082a41"} Nov 02 22:12:33.449738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.449613 4285 scope.go:122] "RemoveContainer" containerID="4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e" Nov 02 22:12:33.460643 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.460619 4285 scope.go:122] "RemoveContainer" containerID="4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e" Nov 02 22:12:33.460954 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:33.460929 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e\": container with ID starting with 4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e not found: ID does not exist" containerID="4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e" Nov 02 22:12:33.461035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.460968 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e"} err="failed to get container status \"4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e\": rpc error: code = NotFound desc = could not find container \"4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e\": container with ID starting with 4bcda706b3a1cbb4c1e84f199f39460ac3723f3f1bbe388bedde7f8591eda69e not found: ID does not exist" Nov 02 22:12:33.650239 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.629582 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn"] Nov 02 22:12:33.673705 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:33.673668 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d1c62e-565c4977bd-cq6dn"] Nov 02 22:12:34.121292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:34.121251 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:34.121675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:34.121314 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:34.463635 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:34.463595 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:34.623941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:34.623901 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:34.814942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:34.814866 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:35.174695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:35.174602 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d8a917-ecae-4593-b818-7a296d45f8fd" path="/var/lib/kubelet/pods/65d8a917-ecae-4593-b818-7a296d45f8fd/volumes" Nov 02 22:12:35.224029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:35.223985 4285 patch_prober.go:29] interesting pod/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:35.224255 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:35.224053 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:35.285639 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:35.285606 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:36.776141 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:36.776089 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h"] Nov 02 22:12:36.776612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:36.776388 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" containerID="cri-o://492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736" gracePeriod=30 Nov 02 22:12:37.078461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.078421 4285 patch_prober.go:29] interesting pod/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:37.078602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.078480 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:37.430833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.430794 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 22:12:37.493454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.493422 4285 generic.go:356] "Generic (PLEG): container finished" podID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerID="d48a390937d235b1ad5c1e16d224d16a0c739aba410e93164c7c60d76e42dd43" exitCode=0 Nov 02 22:12:37.493655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.493497 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" event={"ID":"efc0981a-7ee2-4396-ab57-5a56d7d16323","Type":"ContainerDied","Data":"d48a390937d235b1ad5c1e16d224d16a0c739aba410e93164c7c60d76e42dd43"} Nov 02 22:12:37.495568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.495544 4285 generic.go:356] "Generic (PLEG): container finished" podID="65c76022-a683-4360-9d08-24234e95257d" containerID="2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff" exitCode=0 Nov 02 22:12:37.495666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.495613 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" event={"ID":"65c76022-a683-4360-9d08-24234e95257d","Type":"ContainerDied","Data":"2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff"} Nov 02 22:12:37.495666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.495655 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" event={"ID":"65c76022-a683-4360-9d08-24234e95257d","Type":"ContainerDied","Data":"c793161349528eedb1c808abccce7eb61737d427049d28157fea1aa10e2e3105"} Nov 02 22:12:37.495744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.495677 4285 scope.go:122] "RemoveContainer" containerID="2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff" Nov 02 22:12:37.495803 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.495791 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6" Nov 02 22:12:37.498704 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.498683 4285 generic.go:356] "Generic (PLEG): container finished" podID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerID="fe3245f72fbc4c718cccd6ba62f759baf91f0f4a316d8466b8cf8df1c96c89a1" exitCode=0 Nov 02 22:12:37.498863 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.498839 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" event={"ID":"9bc3b705-119e-4777-a33c-985aa45bd9b8","Type":"ContainerDied","Data":"fe3245f72fbc4c718cccd6ba62f759baf91f0f4a316d8466b8cf8df1c96c89a1"} Nov 02 22:12:37.508782 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.508761 4285 scope.go:122] "RemoveContainer" containerID="8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee" Nov 02 22:12:37.510025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.510007 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:12:37.518778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.518761 4285 scope.go:122] "RemoveContainer" containerID="2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff" Nov 02 22:12:37.519060 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:37.519026 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff\": container with ID starting with 2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff not found: ID does not exist" containerID="2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff" Nov 02 22:12:37.519139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.519073 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff"} err="failed to get container status \"2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff\": rpc error: code = NotFound desc = could not find container \"2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff\": container with ID starting with 2d2aa69eaae11613de1335e4a09f75802b5c0d0ab29faf0c3bedcb6279d4c4ff not found: ID does not exist" Nov 02 22:12:37.519139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.519098 4285 scope.go:122] "RemoveContainer" containerID="8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee" Nov 02 22:12:37.519394 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:37.519373 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee\": container with ID starting with 8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee not found: ID does not exist" containerID="8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee" Nov 02 22:12:37.519457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.519401 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee"} err="failed to get container status \"8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee\": rpc error: code = NotFound desc = could not find container \"8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee\": container with ID starting with 8b264ba8b533c34a7455132665f783579ac5553adc8d01b4c956b9a5c3c53cee not found: ID does not exist" Nov 02 22:12:37.539802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.539781 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:12:37.586992 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.586805 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.586992 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.586923 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.587539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.587052 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.589018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.588948 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.589779 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.589742 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:37.590067 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590039 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.590209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590088 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.590209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590175 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.590344 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590267 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.590434 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590416 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.590511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590465 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.590576 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590508 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs\") pod \"65c76022-a683-4360-9d08-24234e95257d\" (UID: \"65c76022-a683-4360-9d08-24234e95257d\") " Nov 02 22:12:37.590987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590682 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.590987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590744 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.590987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.590931 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.591266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591024 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591475 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c76022-a683-4360-9d08-24234e95257d-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591498 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591509 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591518 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c76022-a683-4360-9d08-24234e95257d-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591528 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591540 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.591563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.591551 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.593449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.593421 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.594110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.594085 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t" pod "65c76022-a683-4360-9d08-24234e95257d" (UID: "65c76022-a683-4360-9d08-24234e95257d"). InnerVolumeSpecName "kube-api-access-c4w5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.692213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692177 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.692396 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692237 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692396 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692339 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692396 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692376 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.692569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692470 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692497 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692516 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.692569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692549 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692549 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.692830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692604 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.692830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692637 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.692830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692665 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.692830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692725 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692755 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.692830 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692818 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692922 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692934 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692948 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.692980 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693011 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693036 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token\") pod \"9bc3b705-119e-4777-a33c-985aa45bd9b8\" (UID: \"9bc3b705-119e-4777-a33c-985aa45bd9b8\") " Nov 02 22:12:37.693143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693068 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs\") pod \"efc0981a-7ee2-4396-ab57-5a56d7d16323\" (UID: \"efc0981a-7ee2-4396-ab57-5a56d7d16323\") " Nov 02 22:12:37.693516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693151 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693269 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.693516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693372 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693479 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693720 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693667 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.693836 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693811 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.693967 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693944 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693974 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.693991 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694005 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694018 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694032 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc0981a-7ee2-4396-ab57-5a56d7d16323-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694045 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694058 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694072 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694086 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc3b705-119e-4777-a33c-985aa45bd9b8-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694100 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4w5t\" (UniqueName: \"kubernetes.io/projected/65c76022-a683-4360-9d08-24234e95257d-kube-api-access-c4w5t\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.694279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.694117 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c76022-a683-4360-9d08-24234e95257d-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.696786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.695625 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.696786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.696187 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:37.696786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.696236 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.696786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.696285 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:37.697491 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.697463 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "kube-api-access-4n2h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.697658 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.697621 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:37.697855 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.697821 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token" pod "9bc3b705-119e-4777-a33c-985aa45bd9b8" (UID: "9bc3b705-119e-4777-a33c-985aa45bd9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.698062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.698032 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7" pod "efc0981a-7ee2-4396-ab57-5a56d7d16323" (UID: "efc0981a-7ee2-4396-ab57-5a56d7d16323"). InnerVolumeSpecName "kube-api-access-4m2l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.795851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795815 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.795851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795853 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.796417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795869 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m2l7\" (UniqueName: \"kubernetes.io/projected/efc0981a-7ee2-4396-ab57-5a56d7d16323-kube-api-access-4m2l7\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.796417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795886 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n2h6\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-kube-api-access-4n2h6\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.796417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795900 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.796417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795914 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.796417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795928 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc3b705-119e-4777-a33c-985aa45bd9b8-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.796417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.795943 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc0981a-7ee2-4396-ab57-5a56d7d16323-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.943718 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.943611 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6"] Nov 02 22:12:37.947589 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:37.947557 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-5xpb6"] Nov 02 22:12:38.507388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.507355 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" Nov 02 22:12:38.507578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.507355 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz" event={"ID":"efc0981a-7ee2-4396-ab57-5a56d7d16323","Type":"ContainerDied","Data":"09666b120782984a20dfb4ef44c916eabd227652e05cb5984c2e647a6c969aa2"} Nov 02 22:12:38.507578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.507509 4285 scope.go:122] "RemoveContainer" containerID="d48a390937d235b1ad5c1e16d224d16a0c739aba410e93164c7c60d76e42dd43" Nov 02 22:12:38.510206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.510184 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" event={"ID":"9bc3b705-119e-4777-a33c-985aa45bd9b8","Type":"ContainerDied","Data":"02ce81bd9b3aedac6c662c2374680aa53fbcec828cc84e425a3c2246239ef62b"} Nov 02 22:12:38.510328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.510188 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4" Nov 02 22:12:38.520282 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.520256 4285 scope.go:122] "RemoveContainer" containerID="fe3245f72fbc4c718cccd6ba62f759baf91f0f4a316d8466b8cf8df1c96c89a1" Nov 02 22:12:38.673274 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.673240 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz"] Nov 02 22:12:38.722957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.722926 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-d88qz"] Nov 02 22:12:38.830623 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.830539 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4"] Nov 02 22:12:38.837331 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:38.837301 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2be7cc3-openshift-default-6469c596f-7vcj4"] Nov 02 22:12:39.056278 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.056243 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh"] Nov 02 22:12:39.056471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.056444 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" podUID="9090c019-7e03-4f72-90e1-2969de759f29" containerName="maas-api" containerID="cri-o://0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70" gracePeriod=30 Nov 02 22:12:39.125979 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.125950 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:39.177095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.177068 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c76022-a683-4360-9d08-24234e95257d" path="/var/lib/kubelet/pods/65c76022-a683-4360-9d08-24234e95257d/volumes" Nov 02 22:12:39.177622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.177606 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" path="/var/lib/kubelet/pods/9bc3b705-119e-4777-a33c-985aa45bd9b8/volumes" Nov 02 22:12:39.178090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.178075 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" path="/var/lib/kubelet/pods/efc0981a-7ee2-4396-ab57-5a56d7d16323/volumes" Nov 02 22:12:39.477961 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.477935 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:12:39.520698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.520667 4285 generic.go:356] "Generic (PLEG): container finished" podID="9090c019-7e03-4f72-90e1-2969de759f29" containerID="0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70" exitCode=0 Nov 02 22:12:39.520857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.520724 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" Nov 02 22:12:39.520857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.520740 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" event={"ID":"9090c019-7e03-4f72-90e1-2969de759f29","Type":"ContainerDied","Data":"0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70"} Nov 02 22:12:39.520857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.520793 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh" event={"ID":"9090c019-7e03-4f72-90e1-2969de759f29","Type":"ContainerDied","Data":"314b9b1926ad1ce8c32fe698b503ad1e2318a7cff023ca6349593cf08f6e8c7e"} Nov 02 22:12:39.520857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.520816 4285 scope.go:122] "RemoveContainer" containerID="0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70" Nov 02 22:12:39.532432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.532418 4285 scope.go:122] "RemoveContainer" containerID="0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70" Nov 02 22:12:39.532678 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:39.532660 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70\": container with ID starting with 0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70 not found: ID does not exist" containerID="0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70" Nov 02 22:12:39.532738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.532686 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70"} err="failed to get container status \"0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70\": rpc error: code = NotFound desc = could not find container \"0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70\": container with ID starting with 0a6e6299651869e81002f589dc3b51a3e4555464fe65f3ecfff3c6802338cb70 not found: ID does not exist" Nov 02 22:12:39.541283 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.541260 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs\") pod \"9090c019-7e03-4f72-90e1-2969de759f29\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " Nov 02 22:12:39.541397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.541307 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\") pod \"9090c019-7e03-4f72-90e1-2969de759f29\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " Nov 02 22:12:39.541463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.541415 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca\") pod \"9090c019-7e03-4f72-90e1-2969de759f29\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " Nov 02 22:12:39.541463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.541460 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4\" (UniqueName: \"kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4\") pod \"9090c019-7e03-4f72-90e1-2969de759f29\" (UID: \"9090c019-7e03-4f72-90e1-2969de759f29\") " Nov 02 22:12:39.541814 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.541787 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca" pod "9090c019-7e03-4f72-90e1-2969de759f29" (UID: "9090c019-7e03-4f72-90e1-2969de759f29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:39.542063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.542040 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090c019-7e03-4f72-90e1-2969de759f29-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:39.543364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.543338 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls" pod "9090c019-7e03-4f72-90e1-2969de759f29" (UID: "9090c019-7e03-4f72-90e1-2969de759f29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:39.543457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.543399 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs" pod "9090c019-7e03-4f72-90e1-2969de759f29" (UID: "9090c019-7e03-4f72-90e1-2969de759f29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:39.543528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.543471 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4" pod "9090c019-7e03-4f72-90e1-2969de759f29" (UID: "9090c019-7e03-4f72-90e1-2969de759f29"). InnerVolumeSpecName "kube-api-access-bklg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:39.643853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.643816 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:39.643853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.643856 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9090c019-7e03-4f72-90e1-2969de759f29-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:39.644055 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.643874 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklg4\" (UniqueName: \"kubernetes.io/projected/9090c019-7e03-4f72-90e1-2969de759f29-kube-api-access-bklg4\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:39.975249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.975199 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh"] Nov 02 22:12:40.009172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:39.991166 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-587629-767bfffd59-r6pzh"] Nov 02 22:12:40.690547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:40.690511 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:41.040751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.040718 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:41.176313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.176283 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9090c019-7e03-4f72-90e1-2969de759f29" path="/var/lib/kubelet/pods/9090c019-7e03-4f72-90e1-2969de759f29/volumes" Nov 02 22:12:41.234409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.234373 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:41.390829 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.390731 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:41.589213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.589178 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2"] Nov 02 22:12:41.589394 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.589376 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" containerID="cri-o://9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1" gracePeriod=30 Nov 02 22:12:41.616053 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.616023 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:41.788581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:41.781971 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:42.055676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.055604 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 22:12:42.208725 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.208685 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.208920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.208780 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.208920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.208877 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209051 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.208939 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209051 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209006 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209056 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209108 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209158 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209002 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:42.209357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209188 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:42.209357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209232 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert\") pod \"6020c3f1-0598-405e-af0f-90bd7b629e1e\" (UID: \"6020c3f1-0598-405e-af0f-90bd7b629e1e\") " Nov 02 22:12:42.209541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209509 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:42.209726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209598 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:42.210200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.209864 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:42.210200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.210092 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.210200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.210118 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.210200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.210152 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.210200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.210165 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.210200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.210178 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6020c3f1-0598-405e-af0f-90bd7b629e1e-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.212338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.212309 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:42.212446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.212355 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:42.212848 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.212820 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "kube-api-access-4sfdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:42.213326 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.213294 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy" pod "6020c3f1-0598-405e-af0f-90bd7b629e1e" (UID: "6020c3f1-0598-405e-af0f-90bd7b629e1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:42.311626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.311530 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.311626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.311570 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.311626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.311585 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6020c3f1-0598-405e-af0f-90bd7b629e1e-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.311626 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.311613 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfdg\" (UniqueName: \"kubernetes.io/projected/6020c3f1-0598-405e-af0f-90bd7b629e1e-kube-api-access-4sfdg\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:42.551962 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.551925 4285 generic.go:356] "Generic (PLEG): container finished" podID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerID="492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736" exitCode=0 Nov 02 22:12:42.552155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.551972 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" Nov 02 22:12:42.552155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.552017 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" event={"ID":"6020c3f1-0598-405e-af0f-90bd7b629e1e","Type":"ContainerDied","Data":"492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736"} Nov 02 22:12:42.552155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.552072 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h" event={"ID":"6020c3f1-0598-405e-af0f-90bd7b629e1e","Type":"ContainerDied","Data":"3eb47648f4e05628dc741128d3bd11a6ab98e49bca7d901b9f7ebe9282c5768d"} Nov 02 22:12:42.552155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.552098 4285 scope.go:122] "RemoveContainer" containerID="492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736" Nov 02 22:12:42.563061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.563035 4285 scope.go:122] "RemoveContainer" containerID="492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736" Nov 02 22:12:42.563350 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:42.563328 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736\": container with ID starting with 492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736 not found: ID does not exist" containerID="492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736" Nov 02 22:12:42.563425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.563359 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736"} err="failed to get container status \"492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736\": rpc error: code = NotFound desc = could not find container \"492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736\": container with ID starting with 492ab3604ffca6bbe5f15a3483581d67ece041cc3e032c8912c7aa6519879736 not found: ID does not exist" Nov 02 22:12:42.716984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.716948 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h"] Nov 02 22:12:42.766270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:42.766232 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-gq49h"] Nov 02 22:12:43.174515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:43.174480 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" path="/var/lib/kubelet/pods/6020c3f1-0598-405e-af0f-90bd7b629e1e/volumes" Nov 02 22:12:44.303692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:44.303655 4285 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:12:44.304077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:44.303711 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:12:46.006778 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:46.006737 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:12:46.149997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.149965 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:46.872765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.872731 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:12:46.996655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996622 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.996849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996682 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.996849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996727 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.996849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996759 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.997022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996861 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.997022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996896 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.997022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996929 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:46.997022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.996975 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.997022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997012 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.997338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997044 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9\") pod \"b7f04c3b-044c-40a2-a010-cc06273a9103\" (UID: \"b7f04c3b-044c-40a2-a010-cc06273a9103\") " Nov 02 22:12:46.997338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997093 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:46.997338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997187 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:46.997338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997196 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:46.997338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997282 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:46.997813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997788 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:46.997813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997815 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f04c3b-044c-40a2-a010-cc06273a9103-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:46.997975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997826 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:46.997975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997834 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:46.997975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.997843 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:46.999208 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.999185 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:12:46.999338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.999192 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:46.999535 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.999518 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:12:46.999616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:46.999595 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9" pod "b7f04c3b-044c-40a2-a010-cc06273a9103" (UID: "b7f04c3b-044c-40a2-a010-cc06273a9103"). InnerVolumeSpecName "kube-api-access-m6zt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:47.104061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.100962 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:47.104061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.101009 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:47.104061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.101025 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f04c3b-044c-40a2-a010-cc06273a9103-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:47.104061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.101041 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6zt9\" (UniqueName: \"kubernetes.io/projected/b7f04c3b-044c-40a2-a010-cc06273a9103-kube-api-access-m6zt9\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:47.193796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.193766 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:47.371106 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.370987 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:47.586644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.586614 4285 generic.go:356] "Generic (PLEG): container finished" podID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerID="9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1" exitCode=0 Nov 02 22:12:47.586813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.586660 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" event={"ID":"b7f04c3b-044c-40a2-a010-cc06273a9103","Type":"ContainerDied","Data":"9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1"} Nov 02 22:12:47.586813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.586671 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" Nov 02 22:12:47.586813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.586704 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2" event={"ID":"b7f04c3b-044c-40a2-a010-cc06273a9103","Type":"ContainerDied","Data":"776e49a310854581c2c5a33af56248b3e2192f497b20517c607c64f5b05c1ae5"} Nov 02 22:12:47.586813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.586727 4285 scope.go:122] "RemoveContainer" containerID="9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1" Nov 02 22:12:47.596798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.596772 4285 scope.go:122] "RemoveContainer" containerID="9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3" Nov 02 22:12:47.609286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.609257 4285 scope.go:122] "RemoveContainer" containerID="9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1" Nov 02 22:12:47.609598 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:47.609570 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1\": container with ID starting with 9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1 not found: ID does not exist" containerID="9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1" Nov 02 22:12:47.609694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.609610 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1"} err="failed to get container status \"9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1\": rpc error: code = NotFound desc = could not find container \"9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1\": container with ID starting with 9f0886f558155c6191fb4b2957ee6a0e02db7ea9b28a639d51c4834f957484b1 not found: ID does not exist" Nov 02 22:12:47.609694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.609634 4285 scope.go:122] "RemoveContainer" containerID="9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3" Nov 02 22:12:47.609942 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:47.609920 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3\": container with ID starting with 9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3 not found: ID does not exist" containerID="9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3" Nov 02 22:12:47.609994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.609949 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3"} err="failed to get container status \"9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3\": rpc error: code = NotFound desc = could not find container \"9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3\": container with ID starting with 9f2f17a92895fe4daaea1a03e45e4a9ba8097619cbd11106131639b97d1ef5e3 not found: ID does not exist" Nov 02 22:12:47.730049 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.730016 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:47.750421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.750391 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2"] Nov 02 22:12:47.801319 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.801287 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d1c62e-openshift-default-748d64c95-2rvf2"] Nov 02 22:12:47.889517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:47.889488 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:48.060385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:48.060311 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:48.629940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:48.627991 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:49.176200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:49.176168 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" path="/var/lib/kubelet/pods/b7f04c3b-044c-40a2-a010-cc06273a9103/volumes" Nov 02 22:12:51.236843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:51.236811 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:52.858433 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:52.858395 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz"] Nov 02 22:12:53.240480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:53.240422 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:53.386009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:53.385964 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:53.727215 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:53.727174 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:56.214759 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:56.214726 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:57.859182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:57.859139 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" podUID="3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" containerName="payload-processing" containerID="cri-o://17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038" gracePeriod=25 Nov 02 22:12:58.134867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.134836 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:12:58.295383 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.295347 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume\") pod \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " Nov 02 22:12:58.295538 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.295467 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx\" (UniqueName: \"kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx\") pod \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\" (UID: \"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b\") " Nov 02 22:12:58.295736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.295711 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume" pod "3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" (UID: "3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:58.295872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.295854 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:58.297709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.297679 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx" pod "3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" (UID: "3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b"). InnerVolumeSpecName "kube-api-access-bjmpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:58.396749 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.396665 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmpx\" (UniqueName: \"kubernetes.io/projected/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b-kube-api-access-bjmpx\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:12:58.720825 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.720784 4285 generic.go:356] "Generic (PLEG): container finished" podID="3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" containerID="17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038" exitCode=0 Nov 02 22:12:58.721011 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.720992 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" event={"ID":"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b","Type":"ContainerDied","Data":"17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038"} Nov 02 22:12:58.721066 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.721053 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" event={"ID":"3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b","Type":"ContainerDied","Data":"7c6fd2024a4eeef3e2ec62f3e1684ae52c763bfc7df8c9e935b1f46c4779bd7a"} Nov 02 22:12:58.721106 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.721083 4285 scope.go:122] "RemoveContainer" containerID="17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038" Nov 02 22:12:58.721286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.721264 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz" Nov 02 22:12:58.734246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.734225 4285 scope.go:122] "RemoveContainer" containerID="17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038" Nov 02 22:12:58.734537 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:58.734514 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038\": container with ID starting with 17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038 not found: ID does not exist" containerID="17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038" Nov 02 22:12:58.734631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.734542 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038"} err="failed to get container status \"17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038\": rpc error: code = NotFound desc = could not find container \"17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038\": container with ID starting with 17a9e57041608072be6b1bc30aced1d55ef51c3aaf057db4daabedac9735a038 not found: ID does not exist" Nov 02 22:12:58.777574 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:12:58.777542 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e06ba03_0bfc_48c2_be32_be8c4f0cdd0b.slice/crio-7c6fd2024a4eeef3e2ec62f3e1684ae52c763bfc7df8c9e935b1f46c4779bd7a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e06ba03_0bfc_48c2_be32_be8c4f0cdd0b.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:12:58.884525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.884491 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz"] Nov 02 22:12:58.888190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:58.888165 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2a9be7-799c4c9d5-6ltmz"] Nov 02 22:12:59.174603 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:12:59.174525 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" path="/var/lib/kubelet/pods/3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b/volumes" Nov 02 22:13:00.514558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:00.514527 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft"] Nov 02 22:13:00.609677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:00.609638 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh"] Nov 02 22:13:00.886503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:00.886422 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:01.063409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:01.063373 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:01.594950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:01.594911 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:01.744503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:01.744464 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:02.058416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:02.058374 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:05.041389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:05.041356 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:05.517163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:05.517078 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerName="payload-processing" containerID="cri-o://d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e" gracePeriod=25 Nov 02 22:13:05.610559 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:05.610516 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerName="payload-pre-processing" containerID="cri-o://9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb" gracePeriod=25 Nov 02 22:13:06.144612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:06.144560 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:13:06.782151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:06.781777 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:07.165722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:07.165629 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:07.373150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:07.372137 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:07.777738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:07.777692 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:08.154771 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:08.154683 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:13:12.875710 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:12.875675 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:13.379499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:13.379465 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:13.534234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:13.534199 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:13.715821 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:13.715787 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:15.386558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:15.386523 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:15.695271 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:15.695237 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc"] Nov 02 22:13:15.695532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:15.695483 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" containerID="cri-o://546a06302adde7a7004039d0dfe04f9a1789487c57e1f30f953ab0131e48ffda" gracePeriod=30 Nov 02 22:13:16.006372 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:13:16.006280 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:13:16.143691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:16.143658 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:13:18.036256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:18.036218 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:18.154713 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:18.154670 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:13:18.907288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:18.907246 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:19.071305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:19.071273 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:20.913219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:20.913160 4285 generic.go:356] "Generic (PLEG): container finished" podID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerID="546a06302adde7a7004039d0dfe04f9a1789487c57e1f30f953ab0131e48ffda" exitCode=0 Nov 02 22:13:20.913714 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:20.913238 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" event={"ID":"bce9f122-a121-4d82-bc6c-3ff4260d7187","Type":"ContainerDied","Data":"546a06302adde7a7004039d0dfe04f9a1789487c57e1f30f953ab0131e48ffda"} Nov 02 22:13:20.979560 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:20.979490 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:13:21.081112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081080 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081190 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081226 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081262 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081344 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081345 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.081483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081385 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081429 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081476 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081557 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket\") pod \"bce9f122-a121-4d82-bc6c-3ff4260d7187\" (UID: \"bce9f122-a121-4d82-bc6c-3ff4260d7187\") " Nov 02 22:13:21.081736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081658 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.081736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081723 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:21.082019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.081812 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.082095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.082030 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.082247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.082230 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.082318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.082248 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.082318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.082257 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce9f122-a121-4d82-bc6c-3ff4260d7187-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.082318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.082268 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.082318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.082278 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.083938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.083909 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "kube-api-access-vfvgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:21.084056 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.083975 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:21.084150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.084107 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:21.084217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.084150 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo" pod "bce9f122-a121-4d82-bc6c-3ff4260d7187" (UID: "bce9f122-a121-4d82-bc6c-3ff4260d7187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:13:21.183861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.183827 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.184019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.183866 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.184019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.183881 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce9f122-a121-4d82-bc6c-3ff4260d7187-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.184019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.183896 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvgl\" (UniqueName: \"kubernetes.io/projected/bce9f122-a121-4d82-bc6c-3ff4260d7187-kube-api-access-vfvgl\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.922527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.922494 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" Nov 02 22:13:21.922527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.922514 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc" event={"ID":"bce9f122-a121-4d82-bc6c-3ff4260d7187","Type":"ContainerDied","Data":"8de4ef17725e6a81f27c0a73af45359c1803fda8bcddb44e9bacdd001da706c2"} Nov 02 22:13:21.923104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:21.922575 4285 scope.go:122] "RemoveContainer" containerID="546a06302adde7a7004039d0dfe04f9a1789487c57e1f30f953ab0131e48ffda" Nov 02 22:13:22.137286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:22.137250 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc"] Nov 02 22:13:22.184877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:22.184790 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-przjc"] Nov 02 22:13:23.103553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:23.103518 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:23.176751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:23.176714 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" path="/var/lib/kubelet/pods/bce9f122-a121-4d82-bc6c-3ff4260d7187/volumes" Nov 02 22:13:25.893750 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.893722 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 22:13:25.915428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.915405 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 22:13:25.963174 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.963143 4285 generic.go:356] "Generic (PLEG): container finished" podID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerID="d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e" exitCode=0 Nov 02 22:13:25.963295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.963189 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" Nov 02 22:13:25.963295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.963229 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" event={"ID":"e9c858eb-093f-4e85-8338-094bb6743dfa","Type":"ContainerDied","Data":"d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e"} Nov 02 22:13:25.963295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.963290 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft" event={"ID":"e9c858eb-093f-4e85-8338-094bb6743dfa","Type":"ContainerDied","Data":"45a843a8589a0e7df71ce2b1a6c79acc2884c259888b98dee2404be925734d2f"} Nov 02 22:13:25.963459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.963318 4285 scope.go:122] "RemoveContainer" containerID="d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e" Nov 02 22:13:25.964588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.964567 4285 generic.go:356] "Generic (PLEG): container finished" podID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerID="9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb" exitCode=0 Nov 02 22:13:25.964702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.964614 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" Nov 02 22:13:25.964702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.964658 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" event={"ID":"833c8ff0-54b9-4604-8af5-0832ffdc86d9","Type":"ContainerDied","Data":"9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb"} Nov 02 22:13:25.964813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.964717 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh" event={"ID":"833c8ff0-54b9-4604-8af5-0832ffdc86d9","Type":"ContainerDied","Data":"7a96c0037d690f67cb2b6a9204812ff184d4b42e2f03d8836be7393957999139"} Nov 02 22:13:25.972495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.972481 4285 scope.go:122] "RemoveContainer" containerID="d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e" Nov 02 22:13:25.972741 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:13:25.972725 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e\": container with ID starting with d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e not found: ID does not exist" containerID="d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e" Nov 02 22:13:25.972783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.972749 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e"} err="failed to get container status \"d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e\": rpc error: code = NotFound desc = could not find container \"d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e\": container with ID starting with d7ba4129c4ae57d32582c0d68a74142fe6fc86a71aeadce58f8d039678abdd0e not found: ID does not exist" Nov 02 22:13:25.972783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.972765 4285 scope.go:122] "RemoveContainer" containerID="9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb" Nov 02 22:13:25.980872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.980854 4285 scope.go:122] "RemoveContainer" containerID="9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb" Nov 02 22:13:25.981167 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:13:25.981147 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb\": container with ID starting with 9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb not found: ID does not exist" containerID="9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb" Nov 02 22:13:25.981236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:25.981173 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb"} err="failed to get container status \"9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb\": rpc error: code = NotFound desc = could not find container \"9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb\": container with ID starting with 9bfb76e77998b0ed69944d032692edcf7a2fab5494ae5efdbcb83e7cd2d159eb not found: ID does not exist" Nov 02 22:13:26.047300 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.047268 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf\" (UniqueName: \"kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf\") pod \"e9c858eb-093f-4e85-8338-094bb6743dfa\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " Nov 02 22:13:26.047482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.047325 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume\") pod \"e9c858eb-093f-4e85-8338-094bb6743dfa\" (UID: \"e9c858eb-093f-4e85-8338-094bb6743dfa\") " Nov 02 22:13:26.047482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.047441 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r\" (UniqueName: \"kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r\") pod \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " Nov 02 22:13:26.047570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.047556 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume\") pod \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\" (UID: \"833c8ff0-54b9-4604-8af5-0832ffdc86d9\") " Nov 02 22:13:26.047887 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.047744 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume" pod "e9c858eb-093f-4e85-8338-094bb6743dfa" (UID: "e9c858eb-093f-4e85-8338-094bb6743dfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:26.048035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.047940 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume" pod "833c8ff0-54b9-4604-8af5-0832ffdc86d9" (UID: "833c8ff0-54b9-4604-8af5-0832ffdc86d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:26.048228 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.048206 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833c8ff0-54b9-4604-8af5-0832ffdc86d9-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:26.048314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.048233 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c858eb-093f-4e85-8338-094bb6743dfa-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:26.049597 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.049574 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf" pod "e9c858eb-093f-4e85-8338-094bb6743dfa" (UID: "e9c858eb-093f-4e85-8338-094bb6743dfa"). InnerVolumeSpecName "kube-api-access-9n9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:26.049954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.049937 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r" pod "833c8ff0-54b9-4604-8af5-0832ffdc86d9" (UID: "833c8ff0-54b9-4604-8af5-0832ffdc86d9"). InnerVolumeSpecName "kube-api-access-kqf9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:26.149655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.149581 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n9gf\" (UniqueName: \"kubernetes.io/projected/e9c858eb-093f-4e85-8338-094bb6743dfa-kube-api-access-9n9gf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:26.149655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.149608 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqf9r\" (UniqueName: \"kubernetes.io/projected/833c8ff0-54b9-4604-8af5-0832ffdc86d9-kube-api-access-kqf9r\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:26.389166 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.389117 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft"] Nov 02 22:13:26.392536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.392507 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f48c4d-779d94b98f-v4vft"] Nov 02 22:13:26.500209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.500177 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh"] Nov 02 22:13:26.505358 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:26.505329 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f48c4d-7b596bd7c9-zzpgh"] Nov 02 22:13:27.174112 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:27.174076 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" path="/var/lib/kubelet/pods/833c8ff0-54b9-4604-8af5-0832ffdc86d9/volumes" Nov 02 22:13:27.174521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:27.174506 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" path="/var/lib/kubelet/pods/e9c858eb-093f-4e85-8338-094bb6743dfa/volumes" Nov 02 22:13:27.589342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:27.589305 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb"] Nov 02 22:13:27.791448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:27.791413 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:27.967856 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:27.967825 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:28.116565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:28.116511 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:30.439807 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.439772 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:30.586388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.586339 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:30.718155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.718108 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs"] Nov 02 22:13:30.727780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.727751 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" containerName="payload-processing" Nov 02 22:13:30.728043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728024 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" containerName="payload-processing" Nov 02 22:13:30.728228 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728214 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e8e87085-4bd5-42cf-8599-1a916c6bb3be" containerName="payload-processing" Nov 02 22:13:30.728343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728330 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e87085-4bd5-42cf-8599-1a916c6bb3be" containerName="payload-processing" Nov 02 22:13:30.728499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728485 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9090c019-7e03-4f72-90e1-2969de759f29" containerName="maas-api" Nov 02 22:13:30.728583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728572 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9090c019-7e03-4f72-90e1-2969de759f29" containerName="maas-api" Nov 02 22:13:30.728659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728648 4285 cpu_manager.go:414] "containerMap: removing container" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" Nov 02 22:13:30.728736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728726 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" Nov 02 22:13:30.728826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728807 4285 cpu_manager.go:414] "containerMap: removing container" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" Nov 02 22:13:30.728826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728822 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" Nov 02 22:13:30.728826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728842 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728848 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728861 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerName="payload-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728868 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerName="payload-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728883 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728890 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728903 4285 cpu_manager.go:414] "containerMap: removing container" podUID="af54beaf-c72f-40c6-adaa-a122e0d0301b" containerName="main" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728910 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af54beaf-c72f-40c6-adaa-a122e0d0301b" containerName="main" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728930 4285 cpu_manager.go:414] "containerMap: removing container" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728937 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728951 4285 cpu_manager.go:414] "containerMap: removing container" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728958 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728973 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b2c3a9a9-5ffd-4638-9986-36ff5876e38f" containerName="payload-pre-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728980 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c3a9a9-5ffd-4638-9986-36ff5876e38f" containerName="payload-pre-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.728995 4285 cpu_manager.go:414] "containerMap: removing container" podUID="016c3683-73b7-41bd-8e88-4f176dbfbdd6" containerName="payload-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729002 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016c3683-73b7-41bd-8e88-4f176dbfbdd6" containerName="payload-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729019 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b0fee743-28c3-48e4-8fe1-326e69e645b3" containerName="maas-api" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729025 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fee743-28c3-48e4-8fe1-326e69e645b3" containerName="maas-api" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729042 4285 cpu_manager.go:414] "containerMap: removing container" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerName="payload-pre-processing" Nov 02 22:13:30.729047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729049 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerName="payload-pre-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729062 4285 cpu_manager.go:414] "containerMap: removing container" podUID="65d8a917-ecae-4593-b818-7a296d45f8fd" containerName="maas-api" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729069 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d8a917-ecae-4593-b818-7a296d45f8fd" containerName="maas-api" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729275 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c3a9a9-5ffd-4638-9986-36ff5876e38f" containerName="payload-pre-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729294 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="016c3683-73b7-41bd-8e88-4f176dbfbdd6" containerName="payload-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729302 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729312 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fee743-28c3-48e4-8fe1-326e69e645b3" containerName="maas-api" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729324 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a60fab5-e08f-49b8-bf6f-0ed7c1260c10" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729332 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729340 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="833c8ff0-54b9-4604-8af5-0832ffdc86d9" containerName="payload-pre-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729348 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d8a917-ecae-4593-b818-7a296d45f8fd" containerName="maas-api" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729356 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e06ba03-0bfc-48c2-be32-be8c4f0cdd0b" containerName="payload-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729365 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e87085-4bd5-42cf-8599-1a916c6bb3be" containerName="payload-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729376 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9090c019-7e03-4f72-90e1-2969de759f29" containerName="maas-api" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729385 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729393 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc0981a-7ee2-4396-ab57-5a56d7d16323" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729403 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc3b705-119e-4777-a33c-985aa45bd9b8" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729411 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c858eb-093f-4e85-8338-094bb6743dfa" containerName="payload-processing" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729421 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729431 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="af54beaf-c72f-40c6-adaa-a122e0d0301b" containerName="main" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729441 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="6020c3f1-0598-405e-af0f-90bd7b629e1e" containerName="istio-proxy" Nov 02 22:13:30.730026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.729448 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce9f122-a121-4d82-bc6c-3ff4260d7187" containerName="istio-proxy" Nov 02 22:13:30.776152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.752811 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.827487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.827450 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs"] Nov 02 22:13:30.834877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.834839 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7rx\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.834904 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.834983 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.835021 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.835161 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.835294 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835379 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.835340 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835494 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.835471 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.835582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.835536 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.892507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.892475 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:30.936975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.936934 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937068 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937115 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937173 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rx\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937213 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937288 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937324 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937396 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937418 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937465 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937547 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937735 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937811 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.937913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.937896 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.939553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.939525 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.939824 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.939808 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.993095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.993035 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:30.993279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:30.993257 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7rx\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx\") pod \"maas-default-gateway-openshift-default-65c448b547-5zlhs\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:31.086020 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:31.085983 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:31.515561 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:31.515523 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs"] Nov 02 22:13:31.516852 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:13:31.516826 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e889220_94d6_4ee1_a77b_d0fed0c099be.slice/crio-3c1b6e5acadeec40112b45c82d3767b653ced528914be8ecf7064a4cead4b8f5 WatchSource:0}: Error finding container 3c1b6e5acadeec40112b45c82d3767b653ced528914be8ecf7064a4cead4b8f5: Status 404 returned error can't find the container with id 3c1b6e5acadeec40112b45c82d3767b653ced528914be8ecf7064a4cead4b8f5 Nov 02 22:13:31.519421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:31.519390 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:31.519512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:31.519449 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:31.519512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:31.519479 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:31.880544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:31.880345 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:32.027196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.027117 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" event={"ID":"5e889220-94d6-4ee1-a77b-d0fed0c099be","Type":"ContainerStarted","Data":"7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92"} Nov 02 22:13:32.027196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.027192 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" event={"ID":"5e889220-94d6-4ee1-a77b-d0fed0c099be","Type":"ContainerStarted","Data":"3c1b6e5acadeec40112b45c82d3767b653ced528914be8ecf7064a4cead4b8f5"} Nov 02 22:13:32.086357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.086328 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:32.088409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.088377 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:32.088569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.088440 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:32.185197 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.181725 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podStartSLOduration=2.181707103 podStartE2EDuration="2.181707103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:32.145966079 +0000 UTC m=+2351.107516409" watchObservedRunningTime="2026-11-02 22:13:32.181707103 +0000 UTC m=+2351.143257423" Nov 02 22:13:32.589621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:32.589585 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" podUID="d01a427e-c877-44eb-bd0d-5d2012f93a8f" containerName="payload-processing" containerID="cri-o://40fa8dca2769c65ad55239956ab1b05498a483fb9726d916065fb3b5c5ea24b5" gracePeriod=25 Nov 02 22:13:33.053168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.053137 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:13:33.053347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.053171 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" event={"ID":"d01a427e-c877-44eb-bd0d-5d2012f93a8f","Type":"ContainerDied","Data":"40fa8dca2769c65ad55239956ab1b05498a483fb9726d916065fb3b5c5ea24b5"} Nov 02 22:13:33.053347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.053118 4285 generic.go:356] "Generic (PLEG): container finished" podID="d01a427e-c877-44eb-bd0d-5d2012f93a8f" containerID="40fa8dca2769c65ad55239956ab1b05498a483fb9726d916065fb3b5c5ea24b5" exitCode=0 Nov 02 22:13:33.053347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.053320 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" event={"ID":"d01a427e-c877-44eb-bd0d-5d2012f93a8f","Type":"ContainerDied","Data":"2ae1c919e719cd8b7f7055984d4bda037e49daf383f40a6e03bf84aeeba14e0b"} Nov 02 22:13:33.053492 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.053374 4285 scope.go:122] "RemoveContainer" containerID="40fa8dca2769c65ad55239956ab1b05498a483fb9726d916065fb3b5c5ea24b5" Nov 02 22:13:33.087050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.087012 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:33.087228 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.087072 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:33.188750 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.188658 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr\" (UniqueName: \"kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr\") pod \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " Nov 02 22:13:33.188921 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.188804 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume\") pod \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\" (UID: \"d01a427e-c877-44eb-bd0d-5d2012f93a8f\") " Nov 02 22:13:33.189230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.189192 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume" pod "d01a427e-c877-44eb-bd0d-5d2012f93a8f" (UID: "d01a427e-c877-44eb-bd0d-5d2012f93a8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:33.189496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.189471 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d01a427e-c877-44eb-bd0d-5d2012f93a8f-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.191096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.191065 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr" pod "d01a427e-c877-44eb-bd0d-5d2012f93a8f" (UID: "d01a427e-c877-44eb-bd0d-5d2012f93a8f"). InnerVolumeSpecName "kube-api-access-prmxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:33.290652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.290610 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prmxr\" (UniqueName: \"kubernetes.io/projected/d01a427e-c877-44eb-bd0d-5d2012f93a8f-kube-api-access-prmxr\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.573197 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.573155 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:33.966441 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:33.966401 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:34.075926 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.075900 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb" Nov 02 22:13:34.087773 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.087522 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:34.087773 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.087580 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:34.139429 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.139376 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:34.250033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.249942 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb"] Nov 02 22:13:34.253992 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.253963 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-caac72-5bcc59868f-hc4rb"] Nov 02 22:13:34.443678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.443640 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:34.630742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:34.630650 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:35.092156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:35.091081 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:35.092156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:35.091158 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:35.160655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:35.160627 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:35.181145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:35.178075 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01a427e-c877-44eb-bd0d-5d2012f93a8f" path="/var/lib/kubelet/pods/d01a427e-c877-44eb-bd0d-5d2012f93a8f/volumes" Nov 02 22:13:35.471225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:35.465214 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:36.086628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:36.086592 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:36.086783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:36.086671 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:36.693733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:36.693702 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:37.086755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:37.086669 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:37.086755 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:37.086718 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:38.086462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:38.086426 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:38.086854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:38.086478 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:39.086790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:39.086750 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-5zlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 02 22:13:39.087287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:39.086821 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 02 22:13:39.335762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:39.335706 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:39.505034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:39.504992 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:39.703146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:39.702133 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:40.091249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:40.091221 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:40.091702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:40.091510 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:40.092238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:40.092219 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:13:40.714048 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:40.714015 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:40.878194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:40.878160 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:41.044100 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:41.044020 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:42.274209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:42.274172 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:42.442455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:42.442403 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:42.592609 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:42.592503 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:42.747387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:42.747352 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:43.072768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:43.072701 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:43.880595 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:43.880558 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:44.052663 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:44.052619 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:44.608632 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:44.608597 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:46.006707 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:13:46.006672 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:13:48.357723 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:48.357687 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:49.617252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:49.617216 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:51.545444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:51.545408 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:51.721678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:51.721626 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:54.005003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.004972 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw"] Nov 02 22:13:54.005506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.005182 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" podUID="3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" containerName="maas-api" containerID="cri-o://f9ca3af78e4d149e96f7fcb8131fca22211db4a48345d1871eb9823b5ab50ec2" gracePeriod=30 Nov 02 22:13:54.124337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.124305 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:54.299573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.299537 4285 generic.go:356] "Generic (PLEG): container finished" podID="3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" containerID="f9ca3af78e4d149e96f7fcb8131fca22211db4a48345d1871eb9823b5ab50ec2" exitCode=0 Nov 02 22:13:54.299730 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.299648 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" event={"ID":"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd","Type":"ContainerDied","Data":"f9ca3af78e4d149e96f7fcb8131fca22211db4a48345d1871eb9823b5ab50ec2"} Nov 02 22:13:54.325066 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.325038 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:13:54.475183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.472276 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\") pod \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " Nov 02 22:13:54.475183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.472530 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47\" (UniqueName: \"kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47\") pod \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " Nov 02 22:13:54.475183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.472607 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca\") pod \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " Nov 02 22:13:54.475183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.472660 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs\") pod \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\" (UID: \"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd\") " Nov 02 22:13:54.479347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.479300 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca" pod "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" (UID: "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:54.492276 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.492211 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47" pod "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" (UID: "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd"). InnerVolumeSpecName "kube-api-access-fkh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:54.506104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.505618 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls" pod "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" (UID: "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:54.506104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.505878 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs" pod "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" (UID: "3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:54.585935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.585252 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkh47\" (UniqueName: \"kubernetes.io/projected/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-kube-api-access-fkh47\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:54.585935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.585293 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:54.585935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.585313 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:54.585935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:54.585327 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:13:55.307891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:55.307858 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" Nov 02 22:13:55.307891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:55.307885 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw" event={"ID":"3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd","Type":"ContainerDied","Data":"daf448220f361533c6be34932d265b3c2fbd334ec1ccd51832287d9308f512e3"} Nov 02 22:13:55.308454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:55.307952 4285 scope.go:122] "RemoveContainer" containerID="f9ca3af78e4d149e96f7fcb8131fca22211db4a48345d1871eb9823b5ab50ec2" Nov 02 22:13:55.473822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:55.473793 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw"] Nov 02 22:13:55.477018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:55.476996 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-027bdf-d6cbb7b49-2djdw"] Nov 02 22:13:55.867968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:55.867884 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:56.066461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:56.066427 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:56.213572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:56.213530 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:56.761944 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:56.761902 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:57.173603 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:13:57.173529 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" path="/var/lib/kubelet/pods/3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd/volumes" Nov 02 22:14:00.340514 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.340482 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz"] Nov 02 22:14:00.340977 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.340669 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" podUID="bfbd8b2b-c40f-462a-8f5d-31c92e17f210" containerName="maas-api" containerID="cri-o://9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11" gracePeriod=30 Nov 02 22:14:00.466763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.466599 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:00.674903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.674874 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.804030 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca\") pod \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.804201 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls\") pod \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.804327 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct\" (UniqueName: \"kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct\") pod \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.804395 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs\") pod \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\" (UID: \"bfbd8b2b-c40f-462a-8f5d-31c92e17f210\") " Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.806873 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca" pod "bfbd8b2b-c40f-462a-8f5d-31c92e17f210" (UID: "bfbd8b2b-c40f-462a-8f5d-31c92e17f210"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.817282 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs" pod "bfbd8b2b-c40f-462a-8f5d-31c92e17f210" (UID: "bfbd8b2b-c40f-462a-8f5d-31c92e17f210"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.822601 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct" pod "bfbd8b2b-c40f-462a-8f5d-31c92e17f210" (UID: "bfbd8b2b-c40f-462a-8f5d-31c92e17f210"). InnerVolumeSpecName "kube-api-access-z6jct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:00.831760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.829965 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls" pod "bfbd8b2b-c40f-462a-8f5d-31c92e17f210" (UID: "bfbd8b2b-c40f-462a-8f5d-31c92e17f210"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:00.921204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.916092 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:00.921204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.916153 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:00.921204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.916169 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jct\" (UniqueName: \"kubernetes.io/projected/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-kube-api-access-z6jct\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:00.921204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:00.916185 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbd8b2b-c40f-462a-8f5d-31c92e17f210-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:01.349218 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.349177 4285 generic.go:356] "Generic (PLEG): container finished" podID="bfbd8b2b-c40f-462a-8f5d-31c92e17f210" containerID="9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11" exitCode=0 Nov 02 22:14:01.349691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.349250 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" event={"ID":"bfbd8b2b-c40f-462a-8f5d-31c92e17f210","Type":"ContainerDied","Data":"9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11"} Nov 02 22:14:01.349800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.349729 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" event={"ID":"bfbd8b2b-c40f-462a-8f5d-31c92e17f210","Type":"ContainerDied","Data":"ea0c7063b6c547c0c9fbb1f76e6764b0d67a15e377e6af4641f0af2b66c0692a"} Nov 02 22:14:01.349800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.349769 4285 scope.go:122] "RemoveContainer" containerID="9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11" Nov 02 22:14:01.349905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.349306 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz" Nov 02 22:14:01.382419 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.382383 4285 scope.go:122] "RemoveContainer" containerID="9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11" Nov 02 22:14:01.383943 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:01.383913 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11\": container with ID starting with 9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11 not found: ID does not exist" containerID="9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11" Nov 02 22:14:01.384068 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.383951 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11"} err="failed to get container status \"9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11\": rpc error: code = NotFound desc = could not find container \"9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11\": container with ID starting with 9e455f72af146001dee28a05610a49f8ecd21434f451ab9fbbb0aecf7ce24a11 not found: ID does not exist" Nov 02 22:14:01.513897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.513861 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz"] Nov 02 22:14:01.564309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:01.564270 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2a9be7-74958b9d76-49vgz"] Nov 02 22:14:02.088690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:02.088657 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:02.257829 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:02.257793 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:02.449098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:02.449061 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:02.617503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:02.617472 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:02.964887 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:02.964856 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:03.127742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:03.127705 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:03.174957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:03.174918 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfbd8b2b-c40f-462a-8f5d-31c92e17f210" path="/var/lib/kubelet/pods/bfbd8b2b-c40f-462a-8f5d-31c92e17f210/volumes" Nov 02 22:14:04.576367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:04.576285 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:07.594207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:07.594169 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:09.065869 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.065827 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:09.264678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.264642 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:09.383165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.383071 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x"] Nov 02 22:14:09.383332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.383306 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" containerID="cri-o://fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a" gracePeriod=30 Nov 02 22:14:09.420063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.420029 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:09.545188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.545149 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf"] Nov 02 22:14:09.720965 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.720933 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:09.873152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:09.873098 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:12.541271 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:12.541234 4285 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:14:12.541648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:12.541291 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:14:12.739876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:12.739843 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:14.671508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.671434 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:14:14.791888 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.791858 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf"] Nov 02 22:14:14.829615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829585 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.829791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829622 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.829791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829648 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.829791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829768 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.829920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829887 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.829967 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829905 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:14.829967 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.829957 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.830075 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830014 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.830168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830085 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.830168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830144 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy\") pod \"5d5e7c02-59cc-4491-8d41-567112f067aa\" (UID: \"5d5e7c02-59cc-4491-8d41-567112f067aa\") " Nov 02 22:14:14.830290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830009 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:14.830349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830108 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:14.830349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830253 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:14.830349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830316 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:14.830747 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830721 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.830847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830756 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e7c02-59cc-4491-8d41-567112f067aa-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.830847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830772 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.830847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830786 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.830847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.830798 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.831930 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.831907 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:14.832516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.832490 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:14:14.832661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.832643 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:14.832888 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.832862 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6" pod "5d5e7c02-59cc-4491-8d41-567112f067aa" (UID: "5d5e7c02-59cc-4491-8d41-567112f067aa"). InnerVolumeSpecName "kube-api-access-w8tc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:14.903287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.903250 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:14.931946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.931864 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.931946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.931893 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8tc6\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-kube-api-access-w8tc6\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.931946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.931905 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:14.931946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:14.931914 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e7c02-59cc-4491-8d41-567112f067aa-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:15.440546 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.440504 4285 generic.go:356] "Generic (PLEG): container finished" podID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerID="fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a" exitCode=0 Nov 02 22:14:15.440726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.440553 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" Nov 02 22:14:15.440726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.440557 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" event={"ID":"5d5e7c02-59cc-4491-8d41-567112f067aa","Type":"ContainerDied","Data":"fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a"} Nov 02 22:14:15.440726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.440619 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x" event={"ID":"5d5e7c02-59cc-4491-8d41-567112f067aa","Type":"ContainerDied","Data":"0b04046d9021b272aa4c29eda55b482c91484bc08360c492461fc921bd2134b0"} Nov 02 22:14:15.440726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.440644 4285 scope.go:122] "RemoveContainer" containerID="fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a" Nov 02 22:14:15.450509 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.450491 4285 scope.go:122] "RemoveContainer" containerID="1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b" Nov 02 22:14:15.462913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.462890 4285 scope.go:122] "RemoveContainer" containerID="fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a" Nov 02 22:14:15.463248 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:15.463224 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a\": container with ID starting with fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a not found: ID does not exist" containerID="fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a" Nov 02 22:14:15.463346 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.463256 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a"} err="failed to get container status \"fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a\": rpc error: code = NotFound desc = could not find container \"fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a\": container with ID starting with fabdc676efba46b5871cb70fceaf9bd8d321363cb15ae1724d132dd8c94a372a not found: ID does not exist" Nov 02 22:14:15.463346 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.463274 4285 scope.go:122] "RemoveContainer" containerID="1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b" Nov 02 22:14:15.463512 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:15.463497 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b\": container with ID starting with 1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b not found: ID does not exist" containerID="1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b" Nov 02 22:14:15.463559 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.463515 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b"} err="failed to get container status \"1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b\": rpc error: code = NotFound desc = could not find container \"1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b\": container with ID starting with 1aa58ac1ee036c7f8eb4b28e88328a68fa3e46b97a694c43d5992107b6c9679b not found: ID does not exist" Nov 02 22:14:15.603908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.603869 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x"] Nov 02 22:14:15.653674 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.653642 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a9be7-openshift-default-7fb9d5cb6c-2tz2x"] Nov 02 22:14:15.713401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:15.713369 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:16.006450 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:16.006358 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:14:16.563875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:16.563842 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7"] Nov 02 22:14:16.944434 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:16.944396 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:17.095809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:17.095762 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:17.183468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:17.183426 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" path="/var/lib/kubelet/pods/5d5e7c02-59cc-4491-8d41-567112f067aa/volumes" Nov 02 22:14:17.451311 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:17.451275 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:18.986507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:18.986474 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:19.200260 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:19.200222 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:19.349289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:19.349182 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:19.932930 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:19.932898 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:21.564598 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:21.564534 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" containerName="payload-processing" containerID="cri-o://02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa" gracePeriod=25 Nov 02 22:14:21.918100 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:21.918011 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd"] Nov 02 22:14:22.234402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:22.234363 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:22.394154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:22.390848 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:22.546249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:22.546170 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:23.855695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:23.855659 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv"] Nov 02 22:14:23.856206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:23.855894 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" podUID="8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" containerName="maas-api" containerID="cri-o://c381ccd61ca628b7e8859500f5a37275937d41b57f5633dc5d31abf7e02ff955" gracePeriod=30 Nov 02 22:14:23.971527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:23.971498 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:24.113606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.113543 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:14:24.124432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.124401 4285 scope.go:122] "RemoveContainer" containerID="c381ccd61ca628b7e8859500f5a37275937d41b57f5633dc5d31abf7e02ff955" Nov 02 22:14:24.292656 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.292613 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\") pod \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " Nov 02 22:14:24.293447 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.293397 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv\" (UniqueName: \"kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv\") pod \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " Nov 02 22:14:24.293819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.293783 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca\") pod \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " Nov 02 22:14:24.293974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.293956 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs\") pod \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\" (UID: \"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9\") " Nov 02 22:14:24.296641 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.296618 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca" pod "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" (UID: "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:24.303364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.302162 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs" pod "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" (UID: "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:24.303503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.303465 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls" pod "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" (UID: "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:24.305670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.305551 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv" pod "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" (UID: "8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9"). InnerVolumeSpecName "kube-api-access-bzdpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:24.406257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.397345 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdpv\" (UniqueName: \"kubernetes.io/projected/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-kube-api-access-bzdpv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:24.406257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.397383 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:24.406257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.397402 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:24.406257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.397415 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:24.518498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.518452 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" event={"ID":"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9","Type":"ContainerDied","Data":"c381ccd61ca628b7e8859500f5a37275937d41b57f5633dc5d31abf7e02ff955"} Nov 02 22:14:24.518694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.518502 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" Nov 02 22:14:24.518694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.518522 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv" event={"ID":"8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9","Type":"ContainerDied","Data":"edcee5ff226267a68d62e84b845e23b0db8558346411a4d3f8ab19900936ff77"} Nov 02 22:14:24.575270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.575228 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" containerID="cri-o://2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4" gracePeriod=45 Nov 02 22:14:24.695561 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.695523 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused" Nov 02 22:14:24.742185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.742111 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv"] Nov 02 22:14:24.745317 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.745287 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-caac72-75bbb79c5b-jpxfv"] Nov 02 22:14:24.923868 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:24.923840 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 22:14:25.006005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.005976 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4\" (UniqueName: \"kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4\") pod \"142239e4-164d-4f9c-a2e0-f4e1537b4796\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " Nov 02 22:14:25.006206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006018 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home\") pod \"142239e4-164d-4f9c-a2e0-f4e1537b4796\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " Nov 02 22:14:25.006206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006080 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir\") pod \"142239e4-164d-4f9c-a2e0-f4e1537b4796\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " Nov 02 22:14:25.006206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006110 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs\" (UniqueName: \"kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs\") pod \"142239e4-164d-4f9c-a2e0-f4e1537b4796\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " Nov 02 22:14:25.006206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006159 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache\") pod \"142239e4-164d-4f9c-a2e0-f4e1537b4796\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " Nov 02 22:14:25.006409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006218 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm\") pod \"142239e4-164d-4f9c-a2e0-f4e1537b4796\" (UID: \"142239e4-164d-4f9c-a2e0-f4e1537b4796\") " Nov 02 22:14:25.006409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006389 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir" pod "142239e4-164d-4f9c-a2e0-f4e1537b4796" (UID: "142239e4-164d-4f9c-a2e0-f4e1537b4796"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.006524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006394 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home" pod "142239e4-164d-4f9c-a2e0-f4e1537b4796" (UID: "142239e4-164d-4f9c-a2e0-f4e1537b4796"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.006524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006473 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache" pod "142239e4-164d-4f9c-a2e0-f4e1537b4796" (UID: "142239e4-164d-4f9c-a2e0-f4e1537b4796"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.006790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006760 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.006790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006782 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.006882 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.006796 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.008368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.008346 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm" pod "142239e4-164d-4f9c-a2e0-f4e1537b4796" (UID: "142239e4-164d-4f9c-a2e0-f4e1537b4796"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:25.008640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.008612 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs" pod "142239e4-164d-4f9c-a2e0-f4e1537b4796" (UID: "142239e4-164d-4f9c-a2e0-f4e1537b4796"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:25.008731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.008638 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4" pod "142239e4-164d-4f9c-a2e0-f4e1537b4796" (UID: "142239e4-164d-4f9c-a2e0-f4e1537b4796"). InnerVolumeSpecName "kube-api-access-5vsb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:25.108172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.108139 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vsb4\" (UniqueName: \"kubernetes.io/projected/142239e4-164d-4f9c-a2e0-f4e1537b4796-kube-api-access-5vsb4\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.108172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.108169 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142239e4-164d-4f9c-a2e0-f4e1537b4796-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.108172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.108180 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142239e4-164d-4f9c-a2e0-f4e1537b4796-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:25.177106 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.177065 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" path="/var/lib/kubelet/pods/8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9/volumes" Nov 02 22:14:25.525979 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.525934 4285 generic.go:356] "Generic (PLEG): container finished" podID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerID="2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4" exitCode=0 Nov 02 22:14:25.526192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.526002 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" Nov 02 22:14:25.526192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.526015 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" event={"ID":"142239e4-164d-4f9c-a2e0-f4e1537b4796","Type":"ContainerDied","Data":"2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4"} Nov 02 22:14:25.526192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.526071 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf" event={"ID":"142239e4-164d-4f9c-a2e0-f4e1537b4796","Type":"ContainerDied","Data":"040453f5c7ad0e7615dd3658b0d05ea48245ee98bef517178d6f46989a2b04f2"} Nov 02 22:14:25.526192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.526094 4285 scope.go:122] "RemoveContainer" containerID="2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4" Nov 02 22:14:25.536233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.536214 4285 scope.go:122] "RemoveContainer" containerID="2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4" Nov 02 22:14:25.536476 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:25.536455 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4\": container with ID starting with 2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4 not found: ID does not exist" containerID="2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4" Nov 02 22:14:25.536542 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.536487 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4"} err="failed to get container status \"2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4\": rpc error: code = NotFound desc = could not find container \"2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4\": container with ID starting with 2a2f0dcd0e11d333bb12006f63ae00ea2ff544e77c96a5c8164a1ac0f0e082e4 not found: ID does not exist" Nov 02 22:14:25.573658 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.573619 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:25.688995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.688964 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf"] Nov 02 22:14:25.692234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.692202 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb8112/premium-w6-fb8112-kserve-697f6d85d8-mp7xf"] Nov 02 22:14:25.853570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:25.853491 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:26.003033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.002997 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:26.172783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.171694 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:26.305759 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.305722 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8"] Nov 02 22:14:26.306003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.305976 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" containerID="cri-o://0ba551c597355f231d7b647970cb1eae10117207e5f287812164294d0fd07829" gracePeriod=30 Nov 02 22:14:26.363277 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.363247 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:26.527601 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.527549 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:26.919147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:26.919033 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerName="payload-pre-processing" containerID="cri-o://45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910" gracePeriod=25 Nov 02 22:14:27.174836 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:27.174751 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" path="/var/lib/kubelet/pods/142239e4-164d-4f9c-a2e0-f4e1537b4796/volumes" Nov 02 22:14:27.642948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:27.642907 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:29.059268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:29.059227 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:29.812687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:29.812631 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" containerID="cri-o://c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f" gracePeriod=45 Nov 02 22:14:29.977602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:29.977562 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused" Nov 02 22:14:30.082887 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.082860 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 22:14:30.182436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182395 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd\" (UniqueName: \"kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd\") pod \"56500d5e-8f5f-4012-8077-e98730a4ab62\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " Nov 02 22:14:30.182621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182445 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir\") pod \"56500d5e-8f5f-4012-8077-e98730a4ab62\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " Nov 02 22:14:30.182621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182561 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home\") pod \"56500d5e-8f5f-4012-8077-e98730a4ab62\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " Nov 02 22:14:30.182752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182660 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs\" (UniqueName: \"kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs\") pod \"56500d5e-8f5f-4012-8077-e98730a4ab62\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " Nov 02 22:14:30.182752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182681 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache\") pod \"56500d5e-8f5f-4012-8077-e98730a4ab62\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " Nov 02 22:14:30.182752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182690 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir" pod "56500d5e-8f5f-4012-8077-e98730a4ab62" (UID: "56500d5e-8f5f-4012-8077-e98730a4ab62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.182752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182750 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm\") pod \"56500d5e-8f5f-4012-8077-e98730a4ab62\" (UID: \"56500d5e-8f5f-4012-8077-e98730a4ab62\") " Nov 02 22:14:30.182968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182837 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home" pod "56500d5e-8f5f-4012-8077-e98730a4ab62" (UID: "56500d5e-8f5f-4012-8077-e98730a4ab62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.182968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.182935 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache" pod "56500d5e-8f5f-4012-8077-e98730a4ab62" (UID: "56500d5e-8f5f-4012-8077-e98730a4ab62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.183368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.183346 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.183368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.183368 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.183555 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.183376 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.184875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.184852 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm" pod "56500d5e-8f5f-4012-8077-e98730a4ab62" (UID: "56500d5e-8f5f-4012-8077-e98730a4ab62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:30.184973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.184879 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs" pod "56500d5e-8f5f-4012-8077-e98730a4ab62" (UID: "56500d5e-8f5f-4012-8077-e98730a4ab62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:30.184973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.184935 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd" pod "56500d5e-8f5f-4012-8077-e98730a4ab62" (UID: "56500d5e-8f5f-4012-8077-e98730a4ab62"). InnerVolumeSpecName "kube-api-access-l9wgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:30.286188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.286150 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56500d5e-8f5f-4012-8077-e98730a4ab62-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.286188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.286186 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56500d5e-8f5f-4012-8077-e98730a4ab62-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.286468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.286200 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9wgd\" (UniqueName: \"kubernetes.io/projected/56500d5e-8f5f-4012-8077-e98730a4ab62-kube-api-access-l9wgd\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:30.562239 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.562200 4285 generic.go:356] "Generic (PLEG): container finished" podID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerID="c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f" exitCode=0 Nov 02 22:14:30.562455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.562267 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" Nov 02 22:14:30.562455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.562289 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" event={"ID":"56500d5e-8f5f-4012-8077-e98730a4ab62","Type":"ContainerDied","Data":"c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f"} Nov 02 22:14:30.562455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.562341 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf" event={"ID":"56500d5e-8f5f-4012-8077-e98730a4ab62","Type":"ContainerDied","Data":"7e286a4276d0c8c4320373588914f4b8ae31d3d3418fdafe11aaca68467340c5"} Nov 02 22:14:30.562455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.562353 4285 scope.go:122] "RemoveContainer" containerID="c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f" Nov 02 22:14:30.574177 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.574152 4285 scope.go:122] "RemoveContainer" containerID="c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f" Nov 02 22:14:30.574611 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:30.574587 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f\": container with ID starting with c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f not found: ID does not exist" containerID="c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f" Nov 02 22:14:30.574736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.574622 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f"} err="failed to get container status \"c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f\": rpc error: code = NotFound desc = could not find container \"c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f\": container with ID starting with c0813ff36cffabb94973054849a1c03108ffe9053791d2b1cbf99140b67fc71f not found: ID does not exist" Nov 02 22:14:30.727353 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.727315 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf"] Nov 02 22:14:30.775105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:30.775072 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-64438f/distinct-w3-64438f-kserve-5c6bcf5955-dfnhf"] Nov 02 22:14:31.076261 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076229 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4"] Nov 02 22:14:31.076873 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076857 4285 cpu_manager.go:414] "containerMap: removing container" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076876 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076888 4285 cpu_manager.go:414] "containerMap: removing container" podUID="bfbd8b2b-c40f-462a-8f5d-31c92e17f210" containerName="maas-api" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076893 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbd8b2b-c40f-462a-8f5d-31c92e17f210" containerName="maas-api" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076906 4285 cpu_manager.go:414] "containerMap: removing container" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076911 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076927 4285 cpu_manager.go:414] "containerMap: removing container" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" Nov 02 22:14:31.076936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076932 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c76022-a683-4360-9d08-24234e95257d" containerName="istio-proxy" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076942 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" containerName="maas-api" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076947 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" containerName="maas-api" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076958 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076963 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f04c3b-044c-40a2-a010-cc06273a9103" containerName="istio-proxy" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076972 4285 cpu_manager.go:414] "containerMap: removing container" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076976 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076984 4285 cpu_manager.go:414] "containerMap: removing container" podUID="8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" containerName="maas-api" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076988 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" containerName="maas-api" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.076997 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d01a427e-c877-44eb-bd0d-5d2012f93a8f" containerName="payload-processing" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077001 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01a427e-c877-44eb-bd0d-5d2012f93a8f" containerName="payload-processing" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077079 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="142239e4-164d-4f9c-a2e0-f4e1537b4796" containerName="main" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077085 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1e9c97-e22c-41d3-890e-71b4d7b2d6e9" containerName="maas-api" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077090 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077098 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01a427e-c877-44eb-bd0d-5d2012f93a8f" containerName="payload-processing" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077107 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" containerName="main" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077114 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbd8b2b-c40f-462a-8f5d-31c92e17f210" containerName="maas-api" Nov 02 22:14:31.077176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.077135 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbaaeca-f6a0-49eb-a5e8-8a1d8fd471fd" containerName="maas-api" Nov 02 22:14:31.081745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.081723 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.098554 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.098531 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.098918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.098559 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.098918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.098609 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6lg9\" (UniqueName: \"kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.098918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.098843 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.133527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.133497 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-af9e7c-serving-cert\"" Nov 02 22:14:31.145875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.145845 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4"] Nov 02 22:14:31.176028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.175987 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56500d5e-8f5f-4012-8077-e98730a4ab62" path="/var/lib/kubelet/pods/56500d5e-8f5f-4012-8077-e98730a4ab62/volumes" Nov 02 22:14:31.200196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.200166 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.200352 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.200222 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.200352 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.200243 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.200352 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.200281 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6lg9\" (UniqueName: \"kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.200542 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:31.200402 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-serving-cert: secret "maas-api-e2e-inf-b-af9e7c-serving-cert" not found Nov 02 22:14:31.200542 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:31.200482 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls podName:3443b75b-8d6e-4388-828c-51520e369633 nodeName:}" failed. No retries permitted until 2026-11-02 22:14:31.700461864 +0000 UTC m=+2410.662012182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls") pod "maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" (UID: "3443b75b-8d6e-4388-828c-51520e369633") : secret "maas-api-e2e-inf-b-af9e7c-serving-cert" not found Nov 02 22:14:31.200919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.200882 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.202875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.202857 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.259817 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.259786 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6lg9\" (UniqueName: \"kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.287172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.287141 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:31.490825 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.490779 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:31.575145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.574407 4285 generic.go:356] "Generic (PLEG): container finished" podID="bfc24004-27d8-41b4-bcca-6270f816e928" containerID="0ba551c597355f231d7b647970cb1eae10117207e5f287812164294d0fd07829" exitCode=0 Nov 02 22:14:31.575145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.574560 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" event={"ID":"bfc24004-27d8-41b4-bcca-6270f816e928","Type":"ContainerDied","Data":"0ba551c597355f231d7b647970cb1eae10117207e5f287812164294d0fd07829"} Nov 02 22:14:31.656211 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.645735 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 22:14:31.711354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711322 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711401 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711458 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711508 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711571 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711586 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.711726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711610 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711690 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711939 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711725 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.711939 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711736 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.711939 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711813 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token\") pod \"bfc24004-27d8-41b4-bcca-6270f816e928\" (UID: \"bfc24004-27d8-41b4-bcca-6270f816e928\") " Nov 02 22:14:31.712099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711975 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.712099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.711974 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.712099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712067 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:31.712302 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712140 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.712425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712397 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.712425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712419 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.712617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712438 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.712617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712452 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.712617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.712466 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfc24004-27d8-41b4-bcca-6270f816e928-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.714434 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.714413 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:31.714578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.714434 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "kube-api-access-c79kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:31.714786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.714760 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:14:31.715598 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.715569 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token" pod "bfc24004-27d8-41b4-bcca-6270f816e928" (UID: "bfc24004-27d8-41b4-bcca-6270f816e928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:31.716101 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.716080 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\") pod \"maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:31.789249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.789180 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:31.813614 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.813578 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.813614 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.813607 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfc24004-27d8-41b4-bcca-6270f816e928-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.813801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.813623 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79kd\" (UniqueName: \"kubernetes.io/projected/bfc24004-27d8-41b4-bcca-6270f816e928-kube-api-access-c79kd\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.813801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.813637 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfc24004-27d8-41b4-bcca-6270f816e928-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:31.994876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:31.994837 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:32.376677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.376648 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4"] Nov 02 22:14:32.378387 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:14:32.378362 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3443b75b_8d6e_4388_828c_51520e369633.slice/crio-e9280b9f5f6fd0d48f3fecd13966ad1275ba44ace4077b299753cd3b84bba231 WatchSource:0}: Error finding container e9280b9f5f6fd0d48f3fecd13966ad1275ba44ace4077b299753cd3b84bba231: Status 404 returned error can't find the container with id e9280b9f5f6fd0d48f3fecd13966ad1275ba44ace4077b299753cd3b84bba231 Nov 02 22:14:32.380222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.380202 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:14:32.583534 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.583502 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" event={"ID":"3443b75b-8d6e-4388-828c-51520e369633","Type":"ContainerStarted","Data":"e9280b9f5f6fd0d48f3fecd13966ad1275ba44ace4077b299753cd3b84bba231"} Nov 02 22:14:32.588586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.588560 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" event={"ID":"bfc24004-27d8-41b4-bcca-6270f816e928","Type":"ContainerDied","Data":"8785c4cfdbc08b9e8c6ed997631ac7b37335a550507b5e4b18a70509c7975d74"} Nov 02 22:14:32.588586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.588580 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8" Nov 02 22:14:32.588764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.588610 4285 scope.go:122] "RemoveContainer" containerID="0ba551c597355f231d7b647970cb1eae10117207e5f287812164294d0fd07829" Nov 02 22:14:32.657573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.657484 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:32.751932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.751897 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8"] Nov 02 22:14:32.801206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.801159 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f48c4d-openshift-default-696bdfbcc5-hnpr8"] Nov 02 22:14:32.824727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.824692 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:32.897329 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.897297 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n"] Nov 02 22:14:32.897591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:32.897544 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" containerID="cri-o://7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315" gracePeriod=30 Nov 02 22:14:33.013148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:33.012970 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:33.173976 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:33.173946 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" path="/var/lib/kubelet/pods/bfc24004-27d8-41b4-bcca-6270f816e928/volumes" Nov 02 22:14:33.604840 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:33.604754 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" event={"ID":"3443b75b-8d6e-4388-828c-51520e369633","Type":"ContainerStarted","Data":"eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12"} Nov 02 22:14:33.605253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:33.604972 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:33.722575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:33.722490 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" podStartSLOduration=1.8504171999999999 podStartE2EDuration="2.72247208s" totalImagesPullingTime="872.05488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:33.718485776 +0000 UTC m=+2412.680036095" watchObservedRunningTime="2026-11-02 22:14:33.72247208 +0000 UTC m=+2412.684022407" Nov 02 22:14:34.033822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:34.033782 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:36.057477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:36.057442 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:36.246148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:36.237019 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:36.793632 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:36.793579 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:37.854690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:37.824782 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.041193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.040960 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.268508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.268482 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 22:14:38.397196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397076 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397175 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397209 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397258 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397294 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397397 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397426 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397499 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.397719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397542 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:38.397719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.397602 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket\") pod \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\" (UID: \"9a87d989-eba7-430a-ac2c-1c45fe0cef7a\") " Nov 02 22:14:38.398349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.398143 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.398972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.398762 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:38.398972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.398783 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:38.399173 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.398996 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:38.399394 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.398478 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:38.403934 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.403900 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:38.411163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.406505 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "kube-api-access-bnv57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:38.411163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.406620 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:38.411163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.410165 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo" pod "9a87d989-eba7-430a-ac2c-1c45fe0cef7a" (UID: "9a87d989-eba7-430a-ac2c-1c45fe0cef7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:14:38.411468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.411421 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499314 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499351 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499368 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv57\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-kube-api-access-bnv57\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499383 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499396 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499409 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499421 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.499480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.499435 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a87d989-eba7-430a-ac2c-1c45fe0cef7a-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:38.579288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.577832 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.730948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.730908 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.838722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.837408 4285 generic.go:356] "Generic (PLEG): container finished" podID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerID="7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315" exitCode=0 Nov 02 22:14:38.838722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.837514 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" event={"ID":"9a87d989-eba7-430a-ac2c-1c45fe0cef7a","Type":"ContainerDied","Data":"7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315"} Nov 02 22:14:38.838722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.837667 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" event={"ID":"9a87d989-eba7-430a-ac2c-1c45fe0cef7a","Type":"ContainerDied","Data":"7e269930390c5aad3283df698a7c0ed4a9e0ce98c4a78036f6ec3c1313d3f24a"} Nov 02 22:14:38.838722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.837703 4285 scope.go:122] "RemoveContainer" containerID="7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315" Nov 02 22:14:38.838722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.837872 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n" Nov 02 22:14:38.873520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.873488 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.873990 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.873760 4285 scope.go:122] "RemoveContainer" containerID="4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b" Nov 02 22:14:38.883441 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.883421 4285 scope.go:122] "RemoveContainer" containerID="7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315" Nov 02 22:14:38.883685 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:38.883664 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315\": container with ID starting with 7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315 not found: ID does not exist" containerID="7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315" Nov 02 22:14:38.883749 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.883693 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315"} err="failed to get container status \"7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315\": rpc error: code = NotFound desc = could not find container \"7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315\": container with ID starting with 7efe1390ab7b00dc8bd30d758ca6f7133799791b0a3f3136ff962767c7755315 not found: ID does not exist" Nov 02 22:14:38.883749 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.883715 4285 scope.go:122] "RemoveContainer" containerID="4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b" Nov 02 22:14:38.883989 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:38.883972 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b\": container with ID starting with 4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b not found: ID does not exist" containerID="4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b" Nov 02 22:14:38.884053 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:38.883993 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b"} err="failed to get container status \"4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b\": rpc error: code = NotFound desc = could not find container \"4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b\": container with ID starting with 4d03bde28b61a70ce7fdea8cbec4a7d837eac00be818ae8fc19aafff0c64262b not found: ID does not exist" Nov 02 22:14:39.005973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:39.005884 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n"] Nov 02 22:14:39.055457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:39.055426 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-caac72-openshift-default-5879bc649f-v8g9n"] Nov 02 22:14:39.059586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:39.059552 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:39.173069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:39.173033 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" path="/var/lib/kubelet/pods/9a87d989-eba7-430a-ac2c-1c45fe0cef7a/volumes" Nov 02 22:14:39.617793 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:39.617764 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:14:41.829510 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.829442 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 22:14:41.869809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.869777 4285 generic.go:356] "Generic (PLEG): container finished" podID="9c7a564d-5b31-49be-8198-2f8103208df1" containerID="02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa" exitCode=0 Nov 02 22:14:41.869963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.869825 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" Nov 02 22:14:41.869963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.869855 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" event={"ID":"9c7a564d-5b31-49be-8198-2f8103208df1","Type":"ContainerDied","Data":"02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa"} Nov 02 22:14:41.869963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.869937 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7" event={"ID":"9c7a564d-5b31-49be-8198-2f8103208df1","Type":"ContainerDied","Data":"2a1a1c1f3db14940270aa8f00405d3c8fa1a12ec9a3d1bfb6b62e00d334c328f"} Nov 02 22:14:41.870075 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.869965 4285 scope.go:122] "RemoveContainer" containerID="02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa" Nov 02 22:14:41.879016 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.878992 4285 scope.go:122] "RemoveContainer" containerID="02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa" Nov 02 22:14:41.879294 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:41.879275 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa\": container with ID starting with 02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa not found: ID does not exist" containerID="02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa" Nov 02 22:14:41.879374 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.879305 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa"} err="failed to get container status \"02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa\": rpc error: code = NotFound desc = could not find container \"02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa\": container with ID starting with 02d00d89ba30ba953620ac8c53a196c6a98272ad41c480c43084e5eb6ccd95fa not found: ID does not exist" Nov 02 22:14:41.944645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.944615 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk\" (UniqueName: \"kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk\") pod \"9c7a564d-5b31-49be-8198-2f8103208df1\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " Nov 02 22:14:41.944838 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.944665 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume\") pod \"9c7a564d-5b31-49be-8198-2f8103208df1\" (UID: \"9c7a564d-5b31-49be-8198-2f8103208df1\") " Nov 02 22:14:41.945170 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.945114 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume" pod "9c7a564d-5b31-49be-8198-2f8103208df1" (UID: "9c7a564d-5b31-49be-8198-2f8103208df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:41.945310 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.945292 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c7a564d-5b31-49be-8198-2f8103208df1-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:41.946850 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:41.946825 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk" pod "9c7a564d-5b31-49be-8198-2f8103208df1" (UID: "9c7a564d-5b31-49be-8198-2f8103208df1"). InnerVolumeSpecName "kube-api-access-hv7lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:42.010647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:42.010611 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:42.063149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:42.060305 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7lk\" (UniqueName: \"kubernetes.io/projected/9c7a564d-5b31-49be-8198-2f8103208df1-kube-api-access-hv7lk\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:42.289056 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:42.289022 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7"] Nov 02 22:14:42.293383 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:42.293358 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb8112-7554759f95-ddgf7"] Nov 02 22:14:43.172995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:43.172961 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" path="/var/lib/kubelet/pods/9c7a564d-5b31-49be-8198-2f8103208df1/volumes" Nov 02 22:14:44.033660 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:44.033620 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:14:44.573960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:44.573926 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:44.871854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:44.871760 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:45.073398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:45.073353 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:45.238772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:45.238736 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:45.252460 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:45.252422 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz"] Nov 02 22:14:45.364195 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:45.364114 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq"] Nov 02 22:14:45.639193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:45.639089 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:45.807613 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:45.807548 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:46.005581 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:46.005542 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:14:46.436680 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.436594 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67"] Nov 02 22:14:46.437226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437202 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" Nov 02 22:14:46.437226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437226 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" Nov 02 22:14:46.437413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437241 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" containerName="payload-processing" Nov 02 22:14:46.437413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437249 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" containerName="payload-processing" Nov 02 22:14:46.437413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437258 4285 cpu_manager.go:414] "containerMap: removing container" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" Nov 02 22:14:46.437413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437265 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" Nov 02 22:14:46.437413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437319 4285 cpu_manager.go:414] "containerMap: removing container" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" Nov 02 22:14:46.437413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437327 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" Nov 02 22:14:46.437712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437464 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc24004-27d8-41b4-bcca-6270f816e928" containerName="istio-proxy" Nov 02 22:14:46.437712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437483 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" Nov 02 22:14:46.437712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437493 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7a564d-5b31-49be-8198-2f8103208df1" containerName="payload-processing" Nov 02 22:14:46.437712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437506 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" Nov 02 22:14:46.437712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.437516 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5e7c02-59cc-4491-8d41-567112f067aa" containerName="istio-proxy" Nov 02 22:14:46.442521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.442491 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.494960 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.494931 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d1a155-serving-cert\"" Nov 02 22:14:46.498409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.498365 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67"] Nov 02 22:14:46.628724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.628682 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.628910 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.628733 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.629005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.628952 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkbv\" (UniqueName: \"kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.629106 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.629077 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.649708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.649670 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:46.754171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.740947 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkbv\" (UniqueName: \"kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.754632 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.754598 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.754861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.754843 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.755435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.755413 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.756044 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.756005 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.758168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.758119 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.759743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.759723 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.803295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.803260 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkbv\" (UniqueName: \"kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv\") pod \"maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:46.815758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.815726 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:46.960406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:46.960376 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:47.058029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.057989 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:47.201977 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.201946 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 22:14:47.261991 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.261958 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4\" (UniqueName: \"kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4\") pod \"ec39698b-c904-4283-86d5-e6b9b5638f2d\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " Nov 02 22:14:47.262224 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.262036 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume\") pod \"ec39698b-c904-4283-86d5-e6b9b5638f2d\" (UID: \"ec39698b-c904-4283-86d5-e6b9b5638f2d\") " Nov 02 22:14:47.263155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.262511 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume" pod "ec39698b-c904-4283-86d5-e6b9b5638f2d" (UID: "ec39698b-c904-4283-86d5-e6b9b5638f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:47.263155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.263111 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39698b-c904-4283-86d5-e6b9b5638f2d-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:47.264659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.264635 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4" pod "ec39698b-c904-4283-86d5-e6b9b5638f2d" (UID: "ec39698b-c904-4283-86d5-e6b9b5638f2d"). InnerVolumeSpecName "kube-api-access-zvsk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:47.364273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.364180 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvsk4\" (UniqueName: \"kubernetes.io/projected/ec39698b-c904-4283-86d5-e6b9b5638f2d-kube-api-access-zvsk4\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:47.451444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.451415 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67"] Nov 02 22:14:47.452675 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:14:47.452648 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2804ab9_431b_45f3_b8ef_23f7c27b2c51.slice/crio-1a38e35159cd1fdeace74d9caf5b7a5125ec21fbfe89eeec503fa3140ee2e927 WatchSource:0}: Error finding container 1a38e35159cd1fdeace74d9caf5b7a5125ec21fbfe89eeec503fa3140ee2e927: Status 404 returned error can't find the container with id 1a38e35159cd1fdeace74d9caf5b7a5125ec21fbfe89eeec503fa3140ee2e927 Nov 02 22:14:47.926730 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.926693 4285 generic.go:356] "Generic (PLEG): container finished" podID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerID="45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910" exitCode=0 Nov 02 22:14:47.927238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.926751 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" Nov 02 22:14:47.927238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.926778 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" event={"ID":"ec39698b-c904-4283-86d5-e6b9b5638f2d","Type":"ContainerDied","Data":"45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910"} Nov 02 22:14:47.927238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.926834 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd" event={"ID":"ec39698b-c904-4283-86d5-e6b9b5638f2d","Type":"ContainerDied","Data":"0a1bcd56582ea09651e2f5bf7eef9448e37233808834e5d048d4d3f36ae4b07c"} Nov 02 22:14:47.927238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.926864 4285 scope.go:122] "RemoveContainer" containerID="45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910" Nov 02 22:14:47.928365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.928338 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" event={"ID":"d2804ab9-431b-45f3-b8ef-23f7c27b2c51","Type":"ContainerStarted","Data":"1a38e35159cd1fdeace74d9caf5b7a5125ec21fbfe89eeec503fa3140ee2e927"} Nov 02 22:14:47.936918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.936896 4285 scope.go:122] "RemoveContainer" containerID="45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910" Nov 02 22:14:47.937306 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:47.937278 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910\": container with ID starting with 45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910 not found: ID does not exist" containerID="45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910" Nov 02 22:14:47.937378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:47.937311 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910"} err="failed to get container status \"45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910\": rpc error: code = NotFound desc = could not find container \"45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910\": container with ID starting with 45357c287dce3462e070d9dbb99c780be2855db3ab6d2ed739b861ee8a673910 not found: ID does not exist" Nov 02 22:14:48.090022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:48.089935 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd"] Nov 02 22:14:48.139395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:48.139353 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-64438f-86f5fc5895-n6bqd"] Nov 02 22:14:48.937711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:48.937639 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" event={"ID":"d2804ab9-431b-45f3-b8ef-23f7c27b2c51","Type":"ContainerStarted","Data":"2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562"} Nov 02 22:14:48.938216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:48.937728 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:49.058975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:49.058920 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" podStartSLOduration=2.240678645 podStartE2EDuration="3.058904793s" totalImagesPullingTime="818.226148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:49.057339751 +0000 UTC m=+2428.018890070" watchObservedRunningTime="2026-11-02 22:14:49.058904793 +0000 UTC m=+2428.020455111" Nov 02 22:14:49.181247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:49.181200 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" path="/var/lib/kubelet/pods/ec39698b-c904-4283-86d5-e6b9b5638f2d/volumes" Nov 02 22:14:50.253093 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.253025 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" podUID="3b403fc0-187f-47a6-83a1-33fc841bca1a" containerName="payload-processing" containerID="cri-o://d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0" gracePeriod=25 Nov 02 22:14:50.365190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.365117 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" podUID="0645dc84-b9fb-47d6-b4e2-87772f483148" containerName="payload-pre-processing" containerID="cri-o://03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd" gracePeriod=25 Nov 02 22:14:50.535904 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.535875 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:14:50.620022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.619978 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume\") pod \"3b403fc0-187f-47a6-83a1-33fc841bca1a\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " Nov 02 22:14:50.620241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.620084 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj\" (UniqueName: \"kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj\") pod \"3b403fc0-187f-47a6-83a1-33fc841bca1a\" (UID: \"3b403fc0-187f-47a6-83a1-33fc841bca1a\") " Nov 02 22:14:50.620451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.620420 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume" pod "3b403fc0-187f-47a6-83a1-33fc841bca1a" (UID: "3b403fc0-187f-47a6-83a1-33fc841bca1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:50.620747 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.620727 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b403fc0-187f-47a6-83a1-33fc841bca1a-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:50.622328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.622299 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj" pod "3b403fc0-187f-47a6-83a1-33fc841bca1a" (UID: "3b403fc0-187f-47a6-83a1-33fc841bca1a"). InnerVolumeSpecName "kube-api-access-qlchj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:50.640594 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.640568 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:14:50.721839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.721799 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume\") pod \"0645dc84-b9fb-47d6-b4e2-87772f483148\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " Nov 02 22:14:50.722020 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.721890 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p\" (UniqueName: \"kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p\") pod \"0645dc84-b9fb-47d6-b4e2-87772f483148\" (UID: \"0645dc84-b9fb-47d6-b4e2-87772f483148\") " Nov 02 22:14:50.722217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.722194 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume" pod "0645dc84-b9fb-47d6-b4e2-87772f483148" (UID: "0645dc84-b9fb-47d6-b4e2-87772f483148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:50.722265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.722240 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlchj\" (UniqueName: \"kubernetes.io/projected/3b403fc0-187f-47a6-83a1-33fc841bca1a-kube-api-access-qlchj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:50.723895 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.723873 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p" pod "0645dc84-b9fb-47d6-b4e2-87772f483148" (UID: "0645dc84-b9fb-47d6-b4e2-87772f483148"). InnerVolumeSpecName "kube-api-access-56k5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:50.823717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.823634 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56k5p\" (UniqueName: \"kubernetes.io/projected/0645dc84-b9fb-47d6-b4e2-87772f483148-kube-api-access-56k5p\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:50.823717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.823667 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0645dc84-b9fb-47d6-b4e2-87772f483148-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:14:50.950251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.950214 4285 generic.go:356] "Generic (PLEG): container finished" podID="3b403fc0-187f-47a6-83a1-33fc841bca1a" containerID="d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0" exitCode=0 Nov 02 22:14:50.950251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.950253 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" Nov 02 22:14:50.950528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.950297 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" event={"ID":"3b403fc0-187f-47a6-83a1-33fc841bca1a","Type":"ContainerDied","Data":"d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0"} Nov 02 22:14:50.950528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.950362 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz" event={"ID":"3b403fc0-187f-47a6-83a1-33fc841bca1a","Type":"ContainerDied","Data":"2fd9f7e21d15a52bee1cb4a2095cb5c9cc69c77d1b1dc579c1a8a38b373f4e87"} Nov 02 22:14:50.950528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.950391 4285 scope.go:122] "RemoveContainer" containerID="d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0" Nov 02 22:14:50.951702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.951680 4285 generic.go:356] "Generic (PLEG): container finished" podID="0645dc84-b9fb-47d6-b4e2-87772f483148" containerID="03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd" exitCode=0 Nov 02 22:14:50.951780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.951743 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" Nov 02 22:14:50.951780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.951764 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" event={"ID":"0645dc84-b9fb-47d6-b4e2-87772f483148","Type":"ContainerDied","Data":"03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd"} Nov 02 22:14:50.951861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.951815 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq" event={"ID":"0645dc84-b9fb-47d6-b4e2-87772f483148","Type":"ContainerDied","Data":"8a633fb6d91d287320d16a8a8bc3a9448b9dea9c8fe7034a865125810ceec3f5"} Nov 02 22:14:50.960469 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.960448 4285 scope.go:122] "RemoveContainer" containerID="d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0" Nov 02 22:14:50.960843 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:50.960821 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0\": container with ID starting with d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0 not found: ID does not exist" containerID="d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0" Nov 02 22:14:50.960946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.960855 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0"} err="failed to get container status \"d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0\": rpc error: code = NotFound desc = could not find container \"d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0\": container with ID starting with d95dbf2e5183cda676e2bbf555182448a2f58b6d733a24bc1e992eabbb2492a0 not found: ID does not exist" Nov 02 22:14:50.960946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.960881 4285 scope.go:122] "RemoveContainer" containerID="03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd" Nov 02 22:14:50.970445 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.970424 4285 scope.go:122] "RemoveContainer" containerID="03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd" Nov 02 22:14:50.970760 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:14:50.970741 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd\": container with ID starting with 03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd not found: ID does not exist" containerID="03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd" Nov 02 22:14:50.970812 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:50.970768 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd"} err="failed to get container status \"03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd\": rpc error: code = NotFound desc = could not find container \"03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd\": container with ID starting with 03d58e750e6fecea908916e4f5106d3e2d56c258c09748436958a943019c4acd not found: ID does not exist" Nov 02 22:14:51.112327 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:51.112237 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq"] Nov 02 22:14:51.116178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:51.116147 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-75577e-688db979b7-qlsbq"] Nov 02 22:14:51.174468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:51.174437 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0645dc84-b9fb-47d6-b4e2-87772f483148" path="/var/lib/kubelet/pods/0645dc84-b9fb-47d6-b4e2-87772f483148/volumes" Nov 02 22:14:51.227242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:51.227207 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz"] Nov 02 22:14:51.230025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:51.230000 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-75577e-85f48f4797-g5gqz"] Nov 02 22:14:51.518748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:51.518706 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:53.173110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:53.173074 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b403fc0-187f-47a6-83a1-33fc841bca1a" path="/var/lib/kubelet/pods/3b403fc0-187f-47a6-83a1-33fc841bca1a/volumes" Nov 02 22:14:54.950039 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:54.950010 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:14:56.215222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:14:56.215186 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:00.188368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.188326 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84"] Nov 02 22:15:00.189086 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189062 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0645dc84-b9fb-47d6-b4e2-87772f483148" containerName="payload-pre-processing" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189087 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0645dc84-b9fb-47d6-b4e2-87772f483148" containerName="payload-pre-processing" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189117 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerName="payload-pre-processing" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189139 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerName="payload-pre-processing" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189178 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3b403fc0-187f-47a6-83a1-33fc841bca1a" containerName="payload-processing" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189186 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b403fc0-187f-47a6-83a1-33fc841bca1a" containerName="payload-processing" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189201 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" Nov 02 22:15:00.189234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189208 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a87d989-eba7-430a-ac2c-1c45fe0cef7a" containerName="istio-proxy" Nov 02 22:15:00.189568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189347 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b403fc0-187f-47a6-83a1-33fc841bca1a" containerName="payload-processing" Nov 02 22:15:00.189568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189367 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0645dc84-b9fb-47d6-b4e2-87772f483148" containerName="payload-pre-processing" Nov 02 22:15:00.189568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.189381 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec39698b-c904-4283-86d5-e6b9b5638f2d" containerName="payload-pre-processing" Nov 02 22:15:00.194291 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.194268 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.205023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.204992 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps"] Nov 02 22:15:00.208981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.208962 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.245819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.245791 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:15:00.341351 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.341311 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmj6h\" (UniqueName: \"kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h\") pod \"maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.341550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.341400 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb896\" (UniqueName: \"kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.341550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.341444 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.341550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.341533 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.369933 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.369896 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps"] Nov 02 22:15:00.386078 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.386049 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84"] Nov 02 22:15:00.442236 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.442212 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmj6h\" (UniqueName: \"kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h\") pod \"maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.442422 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.442297 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb896\" (UniqueName: \"kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.442422 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.442338 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.442422 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.442369 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.443199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.443176 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.443266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.443184 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.498824 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.498789 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmj6h\" (UniqueName: \"kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h\") pod \"maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.498981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.498830 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb896\" (UniqueName: \"kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896\") pod \"maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.509029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.508987 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:00.522250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.522220 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:00.890165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.890136 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84"] Nov 02 22:15:00.891452 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:15:00.891424 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca250868_019a_4b56_aa29_7f8b84b6f3e2.slice/crio-cff801dede7cbba45477f4ece599d7899bfaf51d7116fc985e8fa01c49901c85 WatchSource:0}: Error finding container cff801dede7cbba45477f4ece599d7899bfaf51d7116fc985e8fa01c49901c85: Status 404 returned error can't find the container with id cff801dede7cbba45477f4ece599d7899bfaf51d7116fc985e8fa01c49901c85 Nov 02 22:15:00.899168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.899118 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:00.958762 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:00.958711 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps"] Nov 02 22:15:01.013363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:01.013328 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerStarted","Data":"8dd294b468146a223c56adc16823ab8916d821d4e5338418c98de3240b3052a1"} Nov 02 22:15:01.014937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:01.014905 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" event={"ID":"ca250868-019a-4b56-aa29-7f8b84b6f3e2","Type":"ContainerStarted","Data":"cff801dede7cbba45477f4ece599d7899bfaf51d7116fc985e8fa01c49901c85"} Nov 02 22:15:02.026247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:02.026212 4285 generic.go:356] "Generic (PLEG): container finished" podID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerID="ca4e3891db654082768820deee5d20db15070fa11858c435d2d93fe302ac7e49" exitCode=60 Nov 02 22:15:02.026648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:02.026572 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" event={"ID":"ca250868-019a-4b56-aa29-7f8b84b6f3e2","Type":"ContainerDied","Data":"ca4e3891db654082768820deee5d20db15070fa11858c435d2d93fe302ac7e49"} Nov 02 22:15:02.026783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:02.026766 4285 scope.go:122] "RemoveContainer" containerID="ca4e3891db654082768820deee5d20db15070fa11858c435d2d93fe302ac7e49" Nov 02 22:15:02.033179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:02.030154 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerStarted","Data":"4924cc6ac42375df18674b38fb6e4866b861a9d988aedb163b48d8216c5303d2"} Nov 02 22:15:02.033179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:02.030457 4285 scope.go:122] "RemoveContainer" containerID="4924cc6ac42375df18674b38fb6e4866b861a9d988aedb163b48d8216c5303d2" Nov 02 22:15:03.040859 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.040822 4285 generic.go:356] "Generic (PLEG): container finished" podID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerID="87faa1d2c576c8dc6718fab187ae0af23c203de67d94971b28fa53397fbb56a2" exitCode=60 Nov 02 22:15:03.041307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.040907 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" event={"ID":"ca250868-019a-4b56-aa29-7f8b84b6f3e2","Type":"ContainerDied","Data":"87faa1d2c576c8dc6718fab187ae0af23c203de67d94971b28fa53397fbb56a2"} Nov 02 22:15:03.041307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.040970 4285 scope.go:122] "RemoveContainer" containerID="ca4e3891db654082768820deee5d20db15070fa11858c435d2d93fe302ac7e49" Nov 02 22:15:03.041307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.041299 4285 scope.go:122] "RemoveContainer" containerID="87faa1d2c576c8dc6718fab187ae0af23c203de67d94971b28fa53397fbb56a2" Nov 02 22:15:03.041677 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:03.041654 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84_odh-ai-gateway-infra(ca250868-019a-4b56-aa29-7f8b84b6f3e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" Nov 02 22:15:03.043734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.043712 4285 generic.go:356] "Generic (PLEG): container finished" podID="353dacf1-1460-4620-975f-65c1a48ac794" containerID="4924cc6ac42375df18674b38fb6e4866b861a9d988aedb163b48d8216c5303d2" exitCode=60 Nov 02 22:15:03.043863 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.043793 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerDied","Data":"4924cc6ac42375df18674b38fb6e4866b861a9d988aedb163b48d8216c5303d2"} Nov 02 22:15:03.043863 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.043834 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerStarted","Data":"429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583"} Nov 02 22:15:03.268050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.267999 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" podStartSLOduration=3.26798681 podStartE2EDuration="3.26798681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.266801897 +0000 UTC m=+2442.228352218" watchObservedRunningTime="2026-11-02 22:15:03.26798681 +0000 UTC m=+2442.229537122" Nov 02 22:15:03.717357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.716948 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:03.918954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:03.918912 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:04.051828 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:04.051730 4285 scope.go:122] "RemoveContainer" containerID="87faa1d2c576c8dc6718fab187ae0af23c203de67d94971b28fa53397fbb56a2" Nov 02 22:15:04.052317 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:04.052080 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84_odh-ai-gateway-infra(ca250868-019a-4b56-aa29-7f8b84b6f3e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" Nov 02 22:15:04.054249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:04.053816 4285 generic.go:356] "Generic (PLEG): container finished" podID="353dacf1-1460-4620-975f-65c1a48ac794" containerID="429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583" exitCode=60 Nov 02 22:15:04.054249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:04.053915 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerDied","Data":"429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583"} Nov 02 22:15:04.054249 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:04.053961 4285 scope.go:122] "RemoveContainer" containerID="4924cc6ac42375df18674b38fb6e4866b861a9d988aedb163b48d8216c5303d2" Nov 02 22:15:04.055969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:04.055950 4285 scope.go:122] "RemoveContainer" containerID="429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583" Nov 02 22:15:04.056307 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:04.056281 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps_odh-ai-gateway-infra(353dacf1-1460-4620-975f-65c1a48ac794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" podUID="353dacf1-1460-4620-975f-65c1a48ac794" Nov 02 22:15:05.063668 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:05.063643 4285 scope.go:122] "RemoveContainer" containerID="429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583" Nov 02 22:15:05.064051 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:05.063940 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps_odh-ai-gateway-infra(353dacf1-1460-4620-975f-65c1a48ac794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" podUID="353dacf1-1460-4620-975f-65c1a48ac794" Nov 02 22:15:05.896741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:05.896701 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:06.318407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:06.318375 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:10.831991 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:10.831954 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:12.265545 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:12.265513 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:15.033665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.033631 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g"] Nov 02 22:15:15.039365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.039339 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.086256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.086225 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd"] Nov 02 22:15:15.086480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.086430 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" podUID="b597d323-bab6-4636-8207-b182fbdec288" containerName="maas-api" containerID="cri-o://7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4" gracePeriod=30 Nov 02 22:15:15.133573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.133538 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:15:15.137100 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.137067 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g"] Nov 02 22:15:15.158203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.158172 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.158365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.158231 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.158431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.158418 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pfc\" (UniqueName: \"kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.211335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.211306 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:15.266536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.260062 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pfc\" (UniqueName: \"kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.266536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.260532 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.266536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.260590 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.266536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.261285 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.270746 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.270720 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.342766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.342641 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pfc\" (UniqueName: \"kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc\") pod \"maas-api-revoke-sub-628d3872-zmg4g\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.352937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.352909 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:15.400213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.397519 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 22:15:15.462862 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.462833 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca\") pod \"b597d323-bab6-4636-8207-b182fbdec288\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " Nov 02 22:15:15.463028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.462895 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\") pod \"b597d323-bab6-4636-8207-b182fbdec288\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " Nov 02 22:15:15.463028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.462971 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd\" (UniqueName: \"kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd\") pod \"b597d323-bab6-4636-8207-b182fbdec288\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " Nov 02 22:15:15.463156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.463033 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs\") pod \"b597d323-bab6-4636-8207-b182fbdec288\" (UID: \"b597d323-bab6-4636-8207-b182fbdec288\") " Nov 02 22:15:15.463303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.463278 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca" pod "b597d323-bab6-4636-8207-b182fbdec288" (UID: "b597d323-bab6-4636-8207-b182fbdec288"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:15.463475 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.463459 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b597d323-bab6-4636-8207-b182fbdec288-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:15.465077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.465051 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls" pod "b597d323-bab6-4636-8207-b182fbdec288" (UID: "b597d323-bab6-4636-8207-b182fbdec288"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:15.465197 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.465072 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs" pod "b597d323-bab6-4636-8207-b182fbdec288" (UID: "b597d323-bab6-4636-8207-b182fbdec288"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:15.465440 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.465417 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd" pod "b597d323-bab6-4636-8207-b182fbdec288" (UID: "b597d323-bab6-4636-8207-b182fbdec288"). InnerVolumeSpecName "kube-api-access-2frjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:15.564669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.564611 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:15.564669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.564664 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frjd\" (UniqueName: \"kubernetes.io/projected/b597d323-bab6-4636-8207-b182fbdec288-kube-api-access-2frjd\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:15.564669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.564674 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b597d323-bab6-4636-8207-b182fbdec288-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:15.732574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:15.732540 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g"] Nov 02 22:15:15.733599 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:15:15.733567 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f7ce76_401f_4e76_8f98_3c6d830201e3.slice/crio-5471cd13b49d1e1f599ad3364061f1cb52eed9019aaed28690a9a4fedae62339 WatchSource:0}: Error finding container 5471cd13b49d1e1f599ad3364061f1cb52eed9019aaed28690a9a4fedae62339: Status 404 returned error can't find the container with id 5471cd13b49d1e1f599ad3364061f1cb52eed9019aaed28690a9a4fedae62339 Nov 02 22:15:16.006270 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:16.006187 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:15:16.090388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.090355 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:16.134225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.134189 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" event={"ID":"27f7ce76-401f-4e76-8f98-3c6d830201e3","Type":"ContainerStarted","Data":"aaa4c06d0d5fddd8ac6932616cf957cf563d3505b641a1e116d16417aa51f91b"} Nov 02 22:15:16.134377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.134284 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" event={"ID":"27f7ce76-401f-4e76-8f98-3c6d830201e3","Type":"ContainerStarted","Data":"5471cd13b49d1e1f599ad3364061f1cb52eed9019aaed28690a9a4fedae62339"} Nov 02 22:15:16.135535 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.135507 4285 generic.go:356] "Generic (PLEG): container finished" podID="b597d323-bab6-4636-8207-b182fbdec288" containerID="7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4" exitCode=0 Nov 02 22:15:16.135651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.135560 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" Nov 02 22:15:16.135651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.135554 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" event={"ID":"b597d323-bab6-4636-8207-b182fbdec288","Type":"ContainerDied","Data":"7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4"} Nov 02 22:15:16.135729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.135683 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd" event={"ID":"b597d323-bab6-4636-8207-b182fbdec288","Type":"ContainerDied","Data":"d8c2733b5426567247af16479fda48df4428cb99eae701be2d81b6e09c36f43f"} Nov 02 22:15:16.135729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.135710 4285 scope.go:122] "RemoveContainer" containerID="7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4" Nov 02 22:15:16.164483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.148792 4285 scope.go:122] "RemoveContainer" containerID="7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4" Nov 02 22:15:16.164483 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:16.149067 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4\": container with ID starting with 7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4 not found: ID does not exist" containerID="7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4" Nov 02 22:15:16.164483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.149097 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4"} err="failed to get container status \"7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4\": rpc error: code = NotFound desc = could not find container \"7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4\": container with ID starting with 7bdfde13b881a594d34f2784870028c77722b01b93e59178bda438f8e56f07d4 not found: ID does not exist" Nov 02 22:15:16.270917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.270798 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" podStartSLOduration=2.270779621 podStartE2EDuration="2.270779621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:16.269230059 +0000 UTC m=+2455.230780372" watchObservedRunningTime="2026-11-02 22:15:16.270779621 +0000 UTC m=+2455.232329942" Nov 02 22:15:16.380435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.380402 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd"] Nov 02 22:15:16.384094 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.384059 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-64438f-6c4c4459-l9nrd"] Nov 02 22:15:16.880996 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:16.880962 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:17.143496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.143402 4285 generic.go:356] "Generic (PLEG): container finished" podID="27f7ce76-401f-4e76-8f98-3c6d830201e3" containerID="aaa4c06d0d5fddd8ac6932616cf957cf563d3505b641a1e116d16417aa51f91b" exitCode=0 Nov 02 22:15:17.143920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.143487 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" event={"ID":"27f7ce76-401f-4e76-8f98-3c6d830201e3","Type":"ContainerDied","Data":"aaa4c06d0d5fddd8ac6932616cf957cf563d3505b641a1e116d16417aa51f91b"} Nov 02 22:15:17.157168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.157110 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:17.166514 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.166499 4285 scope.go:122] "RemoveContainer" containerID="429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583" Nov 02 22:15:17.166667 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.166588 4285 scope.go:122] "RemoveContainer" containerID="87faa1d2c576c8dc6718fab187ae0af23c203de67d94971b28fa53397fbb56a2" Nov 02 22:15:17.174014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.173993 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b597d323-bab6-4636-8207-b182fbdec288" path="/var/lib/kubelet/pods/b597d323-bab6-4636-8207-b182fbdec288/volumes" Nov 02 22:15:17.317313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.317280 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:17.784357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:17.784326 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:18.152092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.152065 4285 generic.go:356] "Generic (PLEG): container finished" podID="353dacf1-1460-4620-975f-65c1a48ac794" containerID="81be06326f1774c78f08fbf4cc3b58763ce86fdf0b45be00fa527c5cc4d908e8" exitCode=60 Nov 02 22:15:18.152493 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.152167 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerDied","Data":"81be06326f1774c78f08fbf4cc3b58763ce86fdf0b45be00fa527c5cc4d908e8"} Nov 02 22:15:18.152493 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.152244 4285 scope.go:122] "RemoveContainer" containerID="429046c13ebbd50d854d15bd3a69313cf693654271b4debc0e9f8ed7711af583" Nov 02 22:15:18.152672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.152654 4285 scope.go:122] "RemoveContainer" containerID="81be06326f1774c78f08fbf4cc3b58763ce86fdf0b45be00fa527c5cc4d908e8" Nov 02 22:15:18.153003 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:18.152984 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps_odh-ai-gateway-infra(353dacf1-1460-4620-975f-65c1a48ac794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" podUID="353dacf1-1460-4620-975f-65c1a48ac794" Nov 02 22:15:18.155168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.155148 4285 generic.go:356] "Generic (PLEG): container finished" podID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerID="cf978dc8417f7b08fd0b17d6800a091af1e9e75909adaac94d911fcb8541a2d4" exitCode=60 Nov 02 22:15:18.155287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.155167 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" event={"ID":"ca250868-019a-4b56-aa29-7f8b84b6f3e2","Type":"ContainerDied","Data":"cf978dc8417f7b08fd0b17d6800a091af1e9e75909adaac94d911fcb8541a2d4"} Nov 02 22:15:18.156155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.155578 4285 scope.go:122] "RemoveContainer" containerID="cf978dc8417f7b08fd0b17d6800a091af1e9e75909adaac94d911fcb8541a2d4" Nov 02 22:15:18.156155 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:18.155913 4285 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84_odh-ai-gateway-infra(ca250868-019a-4b56-aa29-7f8b84b6f3e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" Nov 02 22:15:18.180998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.180968 4285 scope.go:122] "RemoveContainer" containerID="87faa1d2c576c8dc6718fab187ae0af23c203de67d94971b28fa53397fbb56a2" Nov 02 22:15:18.314593 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.314562 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:18.400753 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.400726 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc\" (UniqueName: \"kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc\") pod \"27f7ce76-401f-4e76-8f98-3c6d830201e3\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " Nov 02 22:15:18.400938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.400768 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca\") pod \"27f7ce76-401f-4e76-8f98-3c6d830201e3\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " Nov 02 22:15:18.400938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.400807 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca\") pod \"27f7ce76-401f-4e76-8f98-3c6d830201e3\" (UID: \"27f7ce76-401f-4e76-8f98-3c6d830201e3\") " Nov 02 22:15:18.401226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.401194 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca" pod "27f7ce76-401f-4e76-8f98-3c6d830201e3" (UID: "27f7ce76-401f-4e76-8f98-3c6d830201e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:18.401377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.401360 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.402847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.402824 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca" pod "27f7ce76-401f-4e76-8f98-3c6d830201e3" (UID: "27f7ce76-401f-4e76-8f98-3c6d830201e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:18.402981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.402951 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc" pod "27f7ce76-401f-4e76-8f98-3c6d830201e3" (UID: "27f7ce76-401f-4e76-8f98-3c6d830201e3"). InnerVolumeSpecName "kube-api-access-n9pfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:18.502748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.502710 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pfc\" (UniqueName: \"kubernetes.io/projected/27f7ce76-401f-4e76-8f98-3c6d830201e3-kube-api-access-n9pfc\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.502748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:18.502748 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f7ce76-401f-4e76-8f98-3c6d830201e3-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.167650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.167619 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" Nov 02 22:15:19.173338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.173306 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g" event={"ID":"27f7ce76-401f-4e76-8f98-3c6d830201e3","Type":"ContainerDied","Data":"5471cd13b49d1e1f599ad3364061f1cb52eed9019aaed28690a9a4fedae62339"} Nov 02 22:15:19.173482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.173361 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5471cd13b49d1e1f599ad3364061f1cb52eed9019aaed28690a9a4fedae62339" Nov 02 22:15:19.282201 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.282163 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps"] Nov 02 22:15:19.390508 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.390477 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84"] Nov 02 22:15:19.464222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.464149 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:19.522014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.513341 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca\") pod \"353dacf1-1460-4620-975f-65c1a48ac794\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " Nov 02 22:15:19.522014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.513533 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896\" (UniqueName: \"kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896\") pod \"353dacf1-1460-4620-975f-65c1a48ac794\" (UID: \"353dacf1-1460-4620-975f-65c1a48ac794\") " Nov 02 22:15:19.522014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.514860 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca" pod "353dacf1-1460-4620-975f-65c1a48ac794" (UID: "353dacf1-1460-4620-975f-65c1a48ac794"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:19.522014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.518110 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896" pod "353dacf1-1460-4620-975f-65c1a48ac794" (UID: "353dacf1-1460-4620-975f-65c1a48ac794"). InnerVolumeSpecName "kube-api-access-wb896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:19.580770 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.580744 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:19.614549 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.614515 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca\") pod \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " Nov 02 22:15:19.614731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.614663 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h\" (UniqueName: \"kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h\") pod \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\" (UID: \"ca250868-019a-4b56-aa29-7f8b84b6f3e2\") " Nov 02 22:15:19.614889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.614864 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca" pod "ca250868-019a-4b56-aa29-7f8b84b6f3e2" (UID: "ca250868-019a-4b56-aa29-7f8b84b6f3e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:19.615221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.615194 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb896\" (UniqueName: \"kubernetes.io/projected/353dacf1-1460-4620-975f-65c1a48ac794-kube-api-access-wb896\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.615221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.615214 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca250868-019a-4b56-aa29-7f8b84b6f3e2-maas-api-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.615362 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.615227 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353dacf1-1460-4620-975f-65c1a48ac794-maas-api-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.616969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.616945 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h" pod "ca250868-019a-4b56-aa29-7f8b84b6f3e2" (UID: "ca250868-019a-4b56-aa29-7f8b84b6f3e2"). InnerVolumeSpecName "kube-api-access-cmj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:19.716656 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:19.716622 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmj6h\" (UniqueName: \"kubernetes.io/projected/ca250868-019a-4b56-aa29-7f8b84b6f3e2-kube-api-access-cmj6h\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.176449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.176362 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" Nov 02 22:15:20.176449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.176405 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84" event={"ID":"ca250868-019a-4b56-aa29-7f8b84b6f3e2","Type":"ContainerDied","Data":"cff801dede7cbba45477f4ece599d7899bfaf51d7116fc985e8fa01c49901c85"} Nov 02 22:15:20.176978 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.176479 4285 scope.go:122] "RemoveContainer" containerID="cf978dc8417f7b08fd0b17d6800a091af1e9e75909adaac94d911fcb8541a2d4" Nov 02 22:15:20.178173 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.178150 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" Nov 02 22:15:20.178299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.178185 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps" event={"ID":"353dacf1-1460-4620-975f-65c1a48ac794","Type":"ContainerDied","Data":"8dd294b468146a223c56adc16823ab8916d821d4e5338418c98de3240b3052a1"} Nov 02 22:15:20.186033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.186012 4285 scope.go:122] "RemoveContainer" containerID="81be06326f1774c78f08fbf4cc3b58763ce86fdf0b45be00fa527c5cc4d908e8" Nov 02 22:15:20.341794 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.341758 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84"] Nov 02 22:15:20.344851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.344824 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1789c6-29894295-chl84"] Nov 02 22:15:20.453849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.453816 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps"] Nov 02 22:15:20.457717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:20.457684 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-96c052-29894295-nl4ps"] Nov 02 22:15:21.174341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:21.174311 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353dacf1-1460-4620-975f-65c1a48ac794" path="/var/lib/kubelet/pods/353dacf1-1460-4620-975f-65c1a48ac794/volumes" Nov 02 22:15:21.174764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:21.174750 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" path="/var/lib/kubelet/pods/ca250868-019a-4b56-aa29-7f8b84b6f3e2/volumes" Nov 02 22:15:21.799716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:21.799682 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:23.340514 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:23.340476 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:23.499427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:23.499397 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:23.722363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:23.722331 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:23.911204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:23.911176 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:24.120019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:24.119913 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:24.291422 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:24.291394 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:24.452680 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:24.452650 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:24.619016 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:24.618972 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:28.153343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:28.153314 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:28.306552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:28.306521 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:28.816550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:28.816516 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:30.274602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:30.274565 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:30.445832 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:30.445790 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:30.569882 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:30.569798 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj"] Nov 02 22:15:30.570104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:30.570009 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" containerID="cri-o://9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1" gracePeriod=30 Nov 02 22:15:30.610541 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:30.610507 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:31.436081 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:31.436040 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:33.220605 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:33.220539 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:34.882041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:34.882007 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:35.189941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.189854 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:35.342694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.342665 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:35.794582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.794525 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:35.830385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.830363 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 22:15:35.851098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851044 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851117 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851179 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851243 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851238 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851268 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851314 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851369 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851453 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851491 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy\") pod \"700c6c8d-fb45-4029-9204-ad3e41f0657a\" (UID: \"700c6c8d-fb45-4029-9204-ad3e41f0657a\") " Nov 02 22:15:35.851588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851478 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.851588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851537 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.851851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.851792 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:35.852100 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.852041 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.852100 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.852063 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.852100 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.852080 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700c6c8d-fb45-4029-9204-ad3e41f0657a-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.852463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.852409 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.852530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.852481 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.853587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.853544 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "kube-api-access-t5zxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:35.853671 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.853637 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:15:35.853757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.853741 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:35.853947 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.853917 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy" pod "700c6c8d-fb45-4029-9204-ad3e41f0657a" (UID: "700c6c8d-fb45-4029-9204-ad3e41f0657a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:35.953234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.953205 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.953234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.953232 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.953565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.953243 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.953565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.953251 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.953565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.953261 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zxg\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-kube-api-access-t5zxg\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:35.953565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:35.953269 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700c6c8d-fb45-4029-9204-ad3e41f0657a-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:36.292229 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.292197 4285 generic.go:356] "Generic (PLEG): container finished" podID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerID="9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1" exitCode=0 Nov 02 22:15:36.292443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.292250 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" Nov 02 22:15:36.292443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.292295 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" event={"ID":"700c6c8d-fb45-4029-9204-ad3e41f0657a","Type":"ContainerDied","Data":"9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1"} Nov 02 22:15:36.292443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.292352 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj" event={"ID":"700c6c8d-fb45-4029-9204-ad3e41f0657a","Type":"ContainerDied","Data":"68a60343f0f5da1dfa7a865444aa8bc2b5ed01e55368abb282be766d520ccff5"} Nov 02 22:15:36.292443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.292374 4285 scope.go:122] "RemoveContainer" containerID="9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1" Nov 02 22:15:36.302736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.302720 4285 scope.go:122] "RemoveContainer" containerID="3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b" Nov 02 22:15:36.311363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.311345 4285 scope.go:122] "RemoveContainer" containerID="9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1" Nov 02 22:15:36.311612 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:36.311592 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1\": container with ID starting with 9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1 not found: ID does not exist" containerID="9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1" Nov 02 22:15:36.311692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.311619 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1"} err="failed to get container status \"9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1\": rpc error: code = NotFound desc = could not find container \"9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1\": container with ID starting with 9d68f075258850e531bf1496273513ae8fd79cc7200c034ca2cf07a474346db1 not found: ID does not exist" Nov 02 22:15:36.311692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.311636 4285 scope.go:122] "RemoveContainer" containerID="3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b" Nov 02 22:15:36.311864 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:36.311847 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b\": container with ID starting with 3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b not found: ID does not exist" containerID="3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b" Nov 02 22:15:36.311905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.311872 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b"} err="failed to get container status \"3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b\": rpc error: code = NotFound desc = could not find container \"3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b\": container with ID starting with 3007377610fb8f17f235798defcf810c7c2606f1166a0075ff058f569dfac66b not found: ID does not exist" Nov 02 22:15:36.455791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.455766 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj"] Nov 02 22:15:36.505068 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:36.505031 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb8112-openshift-default-6b6fcd5bf7-mmznj"] Nov 02 22:15:37.172494 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:37.172462 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" path="/var/lib/kubelet/pods/700c6c8d-fb45-4029-9204-ad3e41f0657a/volumes" Nov 02 22:15:40.370853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:40.370800 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:41.775186 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:41.775105 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:41.939878 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:41.939815 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:42.041070 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:42.040991 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h"] Nov 02 22:15:42.041287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:42.041261 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" containerID="cri-o://17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b" gracePeriod=30 Nov 02 22:15:42.094602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:42.094564 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:42.534897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:42.534862 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:45.517253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:45.517213 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:45.662734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:45.662706 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:45.994293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:45.994261 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:46.004788 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:46.004760 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:15:46.774941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.774906 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw"] Nov 02 22:15:46.775484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775469 4285 cpu_manager.go:414] "containerMap: removing container" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:46.775529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775486 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:46.775529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775502 4285 cpu_manager.go:414] "containerMap: removing container" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:46.775529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775508 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:46.775529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775521 4285 cpu_manager.go:414] "containerMap: removing container" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" Nov 02 22:15:46.775529 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775527 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775535 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b597d323-bab6-4636-8207-b182fbdec288" containerName="maas-api" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775539 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b597d323-bab6-4636-8207-b182fbdec288" containerName="maas-api" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775549 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775553 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775570 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775578 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775586 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775590 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775600 4285 cpu_manager.go:414] "containerMap: removing container" podUID="27f7ce76-401f-4e76-8f98-3c6d830201e3" containerName="revoke-keys" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775605 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f7ce76-401f-4e76-8f98-3c6d830201e3" containerName="revoke-keys" Nov 02 22:15:46.775682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775684 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775692 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775699 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775704 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b597d323-bab6-4636-8207-b182fbdec288" containerName="maas-api" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775710 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775714 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775721 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775726 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca250868-019a-4b56-aa29-7f8b84b6f3e2" containerName="cleanup" Nov 02 22:15:46.775998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.775731 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f7ce76-401f-4e76-8f98-3c6d830201e3" containerName="revoke-keys" Nov 02 22:15:46.778888 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.778873 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.831796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.831753 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-94a8a3-openshift-default-dockercfg-wmt8x\"" Nov 02 22:15:46.880804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.880763 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.880808 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.880943 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.880989 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.881158 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.881208 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.881274 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.881343 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsmf\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.881389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.881380 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.884046 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.884022 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw"] Nov 02 22:15:46.885631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.885610 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:46.982386 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982353 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982404 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982530 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982623 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsmf\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982692 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982666 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982737 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982747 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982773 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982956 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982833 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.982956 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982864 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.983072 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982967 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.983072 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.982987 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.983343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.983316 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.983440 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.983369 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.985090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.985070 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:46.985267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:46.985249 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:47.039675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.039598 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:47.039847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.039796 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsmf\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf\") pod \"e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:47.092778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.092750 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:47.108399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.108370 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:47.307986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.307957 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:15:47.357536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.357496 4285 generic.go:356] "Generic (PLEG): container finished" podID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerID="17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b" exitCode=0 Nov 02 22:15:47.357703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.357561 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" Nov 02 22:15:47.357703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.357565 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" event={"ID":"6dbf506d-f704-4229-9d75-e2be5bd77925","Type":"ContainerDied","Data":"17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b"} Nov 02 22:15:47.357703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.357638 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h" event={"ID":"6dbf506d-f704-4229-9d75-e2be5bd77925","Type":"ContainerDied","Data":"cdea43179a94c3e7f436f4dc85979a9b76606d65b9995fa6051a681b128eb6f6"} Nov 02 22:15:47.357703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.357667 4285 scope.go:122] "RemoveContainer" containerID="17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b" Nov 02 22:15:47.367272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.367254 4285 scope.go:122] "RemoveContainer" containerID="4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec" Nov 02 22:15:47.375941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.375918 4285 scope.go:122] "RemoveContainer" containerID="17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b" Nov 02 22:15:47.376208 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:47.376190 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b\": container with ID starting with 17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b not found: ID does not exist" containerID="17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b" Nov 02 22:15:47.376269 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.376216 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b"} err="failed to get container status \"17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b\": rpc error: code = NotFound desc = could not find container \"17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b\": container with ID starting with 17b3f908a47459deedd385bc7855117be70d3173a1b7169c81bc31e1bb804b4b not found: ID does not exist" Nov 02 22:15:47.376269 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.376234 4285 scope.go:122] "RemoveContainer" containerID="4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec" Nov 02 22:15:47.376475 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:15:47.376458 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec\": container with ID starting with 4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec not found: ID does not exist" containerID="4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec" Nov 02 22:15:47.376512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.376483 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec"} err="failed to get container status \"4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec\": rpc error: code = NotFound desc = could not find container \"4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec\": container with ID starting with 4fb796d6b1eb7f04a2002cd89f8963b5557a11d01154bee7a53c2df6b9c899ec not found: ID does not exist" Nov 02 22:15:47.386889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.386867 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.386994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.386912 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.386994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.386942 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.386996 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387028 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387100 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387138 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387143 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:47.387295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387170 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387224 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data\") pod \"6dbf506d-f704-4229-9d75-e2be5bd77925\" (UID: \"6dbf506d-f704-4229-9d75-e2be5bd77925\") " Nov 02 22:15:47.387506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387377 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:47.387565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387461 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:47.387565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387529 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:47.387690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387668 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:47.387910 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387889 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.387981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387915 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.387981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387931 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.387981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387946 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.387981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.387960 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf506d-f704-4229-9d75-e2be5bd77925-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.389251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.389228 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:15:47.389251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.389239 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:47.389386 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.389320 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "kube-api-access-v8b4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:47.389386 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.389366 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy" pod "6dbf506d-f704-4229-9d75-e2be5bd77925" (UID: "6dbf506d-f704-4229-9d75-e2be5bd77925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:47.464453 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.464418 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2"] Nov 02 22:15:47.489272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.489242 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.489272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.489269 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.489461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.489280 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8b4s\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-kube-api-access-v8b4s\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.489461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.489290 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbf506d-f704-4229-9d75-e2be5bd77925-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:15:47.523428 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:15:47.523398 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e0ee1c_daab_40ec_9d4e_8ffc6e4ce8c9.slice/crio-de5aba00eab3eb5371fa1439a04e9f6d577cf6f15f587d6fb3a9eefed9ef219e WatchSource:0}: Error finding container de5aba00eab3eb5371fa1439a04e9f6d577cf6f15f587d6fb3a9eefed9ef219e: Status 404 returned error can't find the container with id de5aba00eab3eb5371fa1439a04e9f6d577cf6f15f587d6fb3a9eefed9ef219e Nov 02 22:15:47.523625 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.523600 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw"] Nov 02 22:15:47.526062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.526025 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:47.526206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.526104 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:47.526206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.526176 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:47.583398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.583374 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:47.781350 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.781312 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h"] Nov 02 22:15:47.786727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:47.786702 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-75577e-openshift-default-7d97bf986c-m5n5h"] Nov 02 22:15:48.365382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:48.365284 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" event={"ID":"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9","Type":"ContainerStarted","Data":"5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd"} Nov 02 22:15:48.365382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:48.365344 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" event={"ID":"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9","Type":"ContainerStarted","Data":"de5aba00eab3eb5371fa1439a04e9f6d577cf6f15f587d6fb3a9eefed9ef219e"} Nov 02 22:15:48.480858 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:48.480803 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podStartSLOduration=2.480790072 podStartE2EDuration="2.480790072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:48.479588739 +0000 UTC m=+2487.441139068" watchObservedRunningTime="2026-11-02 22:15:48.480790072 +0000 UTC m=+2487.442340391" Nov 02 22:15:49.093299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:49.093261 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:49.098704 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:49.098678 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:49.173459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:49.173428 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" path="/var/lib/kubelet/pods/6dbf506d-f704-4229-9d75-e2be5bd77925/volumes" Nov 02 22:15:49.373192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:49.373103 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:49.374326 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:49.374306 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:15:50.426847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:50.426810 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:51.274569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:51.274526 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:51.524496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:51.524462 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:51.740818 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:51.740785 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:54.453792 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.453761 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:54.623890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.623852 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:54.840727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.840643 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd"] Nov 02 22:15:54.841403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841383 4285 cpu_manager.go:414] "containerMap: removing container" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" Nov 02 22:15:54.841403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841402 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700c6c8d-fb45-4029-9204-ad3e41f0657a" containerName="istio-proxy" Nov 02 22:15:54.841582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841417 4285 cpu_manager.go:414] "containerMap: removing container" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" Nov 02 22:15:54.841582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841425 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" Nov 02 22:15:54.841582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841486 4285 cpu_manager.go:414] "containerMap: removing container" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:54.841582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841494 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:54.841729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841603 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" Nov 02 22:15:54.841729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.841625 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="353dacf1-1460-4620-975f-65c1a48ac794" containerName="cleanup" Nov 02 22:15:54.845459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.845436 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:54.900212 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.900186 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd"] Nov 02 22:15:54.946406 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.946374 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-94a8a3\"" Nov 02 22:15:54.947431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.946798 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-94a8a3-dockercfg-wl8sk\"" Nov 02 22:15:54.984966 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.984928 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:54.985140 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.985033 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22rn\" (UniqueName: \"kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn\") pod \"payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:54.993212 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.993162 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk"] Nov 02 22:15:54.994007 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.993973 4285 cpu_manager.go:414] "containerMap: removing container" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" Nov 02 22:15:54.994007 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.993996 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" Nov 02 22:15:54.994211 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.994195 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbf506d-f704-4229-9d75-e2be5bd77925" containerName="istio-proxy" Nov 02 22:15:54.997600 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:54.997576 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.053205 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.053171 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk"] Nov 02 22:15:55.086084 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.086050 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvzx\" (UniqueName: \"kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx\") pod \"payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.086299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.086106 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22rn\" (UniqueName: \"kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn\") pod \"payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:55.086379 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.086354 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:55.086484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.086452 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.087309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.087282 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:55.142611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.142535 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22rn\" (UniqueName: \"kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn\") pod \"payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:55.161303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.161274 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:55.187994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.187959 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.188172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.188029 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvzx\" (UniqueName: \"kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx\") pod \"payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.188647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.188627 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.244030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.244000 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvzx\" (UniqueName: \"kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx\") pod \"payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.323987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.323949 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:55.533491 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.533449 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd"] Nov 02 22:15:55.535604 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:15:55.535579 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca086fab_21d1_4fc6_83de_2e36d41841d2.slice/crio-c335169ac1f30d090cddc740431e7feb1867a144c24b302b0138761b7be33fde WatchSource:0}: Error finding container c335169ac1f30d090cddc740431e7feb1867a144c24b302b0138761b7be33fde: Status 404 returned error can't find the container with id c335169ac1f30d090cddc740431e7feb1867a144c24b302b0138761b7be33fde Nov 02 22:15:55.616020 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.615988 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:55.703420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.703393 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk"] Nov 02 22:15:55.705047 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:15:55.705011 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33952884_fe06_4862_9077_7cfbc526fa76.slice/crio-3d7bb997ee4029d1db543cad086ba49b223452031d763ff6b8e539d62762fc62 WatchSource:0}: Error finding container 3d7bb997ee4029d1db543cad086ba49b223452031d763ff6b8e539d62762fc62: Status 404 returned error can't find the container with id 3d7bb997ee4029d1db543cad086ba49b223452031d763ff6b8e539d62762fc62 Nov 02 22:15:55.906169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:55.906073 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:56.064153 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.062562 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:56.418193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.418152 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" event={"ID":"ca086fab-21d1-4fc6-83de-2e36d41841d2","Type":"ContainerStarted","Data":"15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c"} Nov 02 22:15:56.418515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.418486 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:15:56.418596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.418529 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" event={"ID":"ca086fab-21d1-4fc6-83de-2e36d41841d2","Type":"ContainerStarted","Data":"c335169ac1f30d090cddc740431e7feb1867a144c24b302b0138761b7be33fde"} Nov 02 22:15:56.420021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.419996 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" event={"ID":"33952884-fe06-4862-9077-7cfbc526fa76","Type":"ContainerStarted","Data":"d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697"} Nov 02 22:15:56.420174 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.420038 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" event={"ID":"33952884-fe06-4862-9077-7cfbc526fa76","Type":"ContainerStarted","Data":"3d7bb997ee4029d1db543cad086ba49b223452031d763ff6b8e539d62762fc62"} Nov 02 22:15:56.420174 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.420058 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:15:56.531592 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.531531 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" podStartSLOduration=2.531516358 podStartE2EDuration="2.531516358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:56.529468726 +0000 UTC m=+2495.491019080" watchObservedRunningTime="2026-11-02 22:15:56.531516358 +0000 UTC m=+2495.493066675" Nov 02 22:15:56.658659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:15:56.658583 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" podStartSLOduration=2.658562545 podStartE2EDuration="2.658562545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:56.651240281 +0000 UTC m=+2495.612790601" watchObservedRunningTime="2026-11-02 22:15:56.658562545 +0000 UTC m=+2495.620112866" Nov 02 22:16:00.413534 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:00.413468 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:02.485998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.485938 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" containerID="cri-o://354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf" gracePeriod=45 Nov 02 22:16:02.754385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.754321 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 22:16:02.776279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776247 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm\") pod \"fde78196-f98e-444a-9419-25e38531db04\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " Nov 02 22:16:02.776425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776354 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs\" (UniqueName: \"kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs\") pod \"fde78196-f98e-444a-9419-25e38531db04\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " Nov 02 22:16:02.776425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776399 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache\") pod \"fde78196-f98e-444a-9419-25e38531db04\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " Nov 02 22:16:02.776532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776441 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home\") pod \"fde78196-f98e-444a-9419-25e38531db04\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " Nov 02 22:16:02.776532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776483 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9\" (UniqueName: \"kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9\") pod \"fde78196-f98e-444a-9419-25e38531db04\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " Nov 02 22:16:02.776532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776511 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir\") pod \"fde78196-f98e-444a-9419-25e38531db04\" (UID: \"fde78196-f98e-444a-9419-25e38531db04\") " Nov 02 22:16:02.776721 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776681 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home" pod "fde78196-f98e-444a-9419-25e38531db04" (UID: "fde78196-f98e-444a-9419-25e38531db04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.776790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776705 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache" pod "fde78196-f98e-444a-9419-25e38531db04" (UID: "fde78196-f98e-444a-9419-25e38531db04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.776839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776786 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir" pod "fde78196-f98e-444a-9419-25e38531db04" (UID: "fde78196-f98e-444a-9419-25e38531db04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.776973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776956 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.777041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776975 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.777041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.776985 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.778651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.778623 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs" pod "fde78196-f98e-444a-9419-25e38531db04" (UID: "fde78196-f98e-444a-9419-25e38531db04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:02.778651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.778630 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9" pod "fde78196-f98e-444a-9419-25e38531db04" (UID: "fde78196-f98e-444a-9419-25e38531db04"). InnerVolumeSpecName "kube-api-access-npll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:02.778779 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.778682 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm" pod "fde78196-f98e-444a-9419-25e38531db04" (UID: "fde78196-f98e-444a-9419-25e38531db04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:02.877685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.877648 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde78196-f98e-444a-9419-25e38531db04-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.877685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.877684 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npll9\" (UniqueName: \"kubernetes.io/projected/fde78196-f98e-444a-9419-25e38531db04-kube-api-access-npll9\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.877938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:02.877697 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde78196-f98e-444a-9419-25e38531db04-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:16:03.484923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.484878 4285 generic.go:356] "Generic (PLEG): container finished" podID="fde78196-f98e-444a-9419-25e38531db04" containerID="354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf" exitCode=0 Nov 02 22:16:03.485143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.484928 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" Nov 02 22:16:03.485143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.484950 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" event={"ID":"fde78196-f98e-444a-9419-25e38531db04","Type":"ContainerDied","Data":"354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf"} Nov 02 22:16:03.485143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.484990 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" event={"ID":"fde78196-f98e-444a-9419-25e38531db04","Type":"ContainerDied","Data":"6f9a0d2e83eb056907b9ba322e574e3a3cf35d8b98a2f51d2c9ed4f8ec10692b"} Nov 02 22:16:03.485143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.485006 4285 scope.go:122] "RemoveContainer" containerID="354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf" Nov 02 22:16:03.494225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.494019 4285 scope.go:122] "RemoveContainer" containerID="354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf" Nov 02 22:16:03.494485 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:16:03.494361 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf\": container with ID starting with 354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf not found: ID does not exist" containerID="354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf" Nov 02 22:16:03.494485 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.494386 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf"} err="failed to get container status \"354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf\": rpc error: code = NotFound desc = could not find container \"354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf\": container with ID starting with 354b12fa5f816a6771a99b0c06d9d045d75cb3fcb6b872ae340ab97792ff3fbf not found: ID does not exist" Nov 02 22:16:03.647678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.647646 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2"] Nov 02 22:16:03.693841 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.693801 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:03.696972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:03.696948 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-96c052/distinct-2-w5-96c052-kserve-559d8bb5b-zx4p2"] Nov 02 22:16:04.678970 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:04.678933 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:04.859184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:04.849802 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:05.174618 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:05.174576 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde78196-f98e-444a-9419-25e38531db04" path="/var/lib/kubelet/pods/fde78196-f98e-444a-9419-25e38531db04/volumes" Nov 02 22:16:05.417516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:05.417469 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:05.578571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:05.578530 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:06.074192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:06.074150 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:06.318499 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:06.318461 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:06.473251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:06.473209 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:07.009285 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:07.009254 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:07.428929 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:07.428844 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:16:07.429385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:07.429035 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:16:11.303894 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:11.303861 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:12.431708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:12.431665 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:13.324832 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:13.324785 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:13.488675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:13.488637 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:13.674220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:13.674132 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:16.003604 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:16.003549 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:16.005811 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:16:16.005781 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:16:16.413043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:16.412917 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:20.707727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:20.707694 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:23.995252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:23.995218 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:24.165272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:24.165149 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:25.217927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:25.217890 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:25.446679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:25.446647 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:30.097547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:30.097507 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:30.769539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:30.769507 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:31.061226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:31.061087 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:34.747327 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:34.747290 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:34.901053 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:34.901019 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:35.055198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:35.055101 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:39.619464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:39.619425 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.406719 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:40.406687 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.731506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:40.731474 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:44.602313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:44.602280 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:46.004693 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:16:46.004658 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:16:49.339976 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:49.339910 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:53.997616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:53.997566 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:58.705737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:16:58.705703 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:00.838085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:00.838020 4285 generic.go:356] "Generic (PLEG): container finished" podID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerID="5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd" exitCode=0 Nov 02 22:17:00.838476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:00.838095 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" event={"ID":"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9","Type":"ContainerDied","Data":"5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd"} Nov 02 22:17:00.838520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:00.838510 4285 scope.go:122] "RemoveContainer" containerID="5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd" Nov 02 22:17:00.839918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:00.839888 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:00.840008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:00.839962 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:00.840008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:00.839995 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:01.852282 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:01.852246 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" event={"ID":"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9","Type":"ContainerStarted","Data":"2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962"} Nov 02 22:17:02.093344 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:02.093312 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:17:02.094257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:02.094234 4285 patch_prober.go:29] interesting pod/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:17:02.094348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:02.094284 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:17:03.094086 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:03.094046 4285 patch_prober.go:29] interesting pod/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:17:03.094494 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:03.094091 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:17:03.377897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:03.377797 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:04.093040 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:04.093009 4285 patch_prober.go:29] interesting pod/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:17:04.093241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:04.093057 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:17:04.373750 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:04.373654 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:17:05.093831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:05.093799 4285 patch_prober.go:29] interesting pod/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:17:05.094041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:05.093853 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:17:06.093795 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:06.093751 4285 patch_prober.go:29] interesting pod/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:17:06.094394 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:06.094364 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 22:17:07.097455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:07.097426 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:17:07.098399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:07.098374 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:17:08.097119 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:08.097082 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:11.110074 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:11.110042 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:11.265632 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:11.265603 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:12.903326 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:12.903292 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:16.005584 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:17:16.005543 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:17:16.752442 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:16.752414 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:17.615176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:17.615133 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:22.314041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:22.313967 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:27.324188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:27.324141 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:27.916253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:27.916223 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:28.523054 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:28.523023 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:30.814685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:30.814652 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:31.441881 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:31.441850 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:32.979041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:32.979003 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:33.252188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:33.252156 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:33.406761 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:33.406726 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:33.882745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:33.882717 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:38.104675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:38.104645 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.033342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:40.033302 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.167348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:40.167316 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.336656 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:40.336579 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878"] Nov 02 22:17:40.336827 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:40.336782 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" containerID="cri-o://0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a" gracePeriod=30 Nov 02 22:17:40.364459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:40.364424 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:42.901476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:42.901442 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:45.594909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.594879 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 22:17:45.722336 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722301 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722377 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722449 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722500 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722525 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722566 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722602 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.722690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722618 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722649 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.722943 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722709 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data\") pod \"847e7104-aadd-4903-835f-d71b8dce4beb\" (UID: \"847e7104-aadd-4903-835f-d71b8dce4beb\") " Nov 02 22:17:45.723000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722931 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.723000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.722948 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:45.723187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.723138 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.723187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.723153 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.723297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.723222 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.723297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.723242 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/847e7104-aadd-4903-835f-d71b8dce4beb-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.723297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.723258 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.725030 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.725003 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:17:45.725292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.725265 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:17:45.725433 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.725298 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:45.725433 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.725352 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892" pod "847e7104-aadd-4903-835f-d71b8dce4beb" (UID: "847e7104-aadd-4903-835f-d71b8dce4beb"). InnerVolumeSpecName "kube-api-access-d8892". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:45.824709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.824683 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.824709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.824707 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.824709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.824717 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.824995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.824726 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/847e7104-aadd-4903-835f-d71b8dce4beb-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.824995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.824735 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8892\" (UniqueName: \"kubernetes.io/projected/847e7104-aadd-4903-835f-d71b8dce4beb-kube-api-access-d8892\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:45.824995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:45.824745 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/847e7104-aadd-4903-835f-d71b8dce4beb-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:17:46.005558 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:17:46.005485 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:17:46.113846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.113812 4285 generic.go:356] "Generic (PLEG): container finished" podID="847e7104-aadd-4903-835f-d71b8dce4beb" containerID="0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a" exitCode=0 Nov 02 22:17:46.113999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.113854 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" Nov 02 22:17:46.113999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.113900 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" event={"ID":"847e7104-aadd-4903-835f-d71b8dce4beb","Type":"ContainerDied","Data":"0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a"} Nov 02 22:17:46.113999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.113955 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878" event={"ID":"847e7104-aadd-4903-835f-d71b8dce4beb","Type":"ContainerDied","Data":"7dca9ff0ee7b3bc3a41ece74b18bf75a7788b5461afd0eba64b2cb8b553fc8fe"} Nov 02 22:17:46.113999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.113985 4285 scope.go:122] "RemoveContainer" containerID="0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a" Nov 02 22:17:46.127975 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.127637 4285 scope.go:122] "RemoveContainer" containerID="0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a" Nov 02 22:17:46.128866 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:17:46.128801 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a\": container with ID starting with 0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a not found: ID does not exist" containerID="0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a" Nov 02 22:17:46.128866 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.128839 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a"} err="failed to get container status \"0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a\": rpc error: code = NotFound desc = could not find container \"0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a\": container with ID starting with 0b630ac97d909540e1c70efefd1850844f076756b8e1713d9ef981c721af529a not found: ID does not exist" Nov 02 22:17:46.277022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.276945 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878"] Nov 02 22:17:46.328063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:46.328032 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-96c052-openshift-default-86677f975c-d6878"] Nov 02 22:17:47.172669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:47.172633 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" path="/var/lib/kubelet/pods/847e7104-aadd-4903-835f-d71b8dce4beb/volumes" Nov 02 22:17:47.599720 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:47.599686 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:52.299175 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:52.299121 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:56.999534 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:17:56.999501 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:01.704596 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:01.704563 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:06.419290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:06.419253 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:11.099283 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:11.099245 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:11.695299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:11.695262 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:15.973735 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:15.973702 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:16.005456 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:18:16.005420 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:18:17.148630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:17.148597 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:20.611089 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:20.611060 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:25.309695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:25.309660 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:29.999402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:29.999369 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:34.703355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:34.703247 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:39.414198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:39.414159 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:44.121525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:44.121483 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:46.006814 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:18:46.006780 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:18:48.816722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:48.816688 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:49.341544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:49.341513 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:53.534136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:53.534091 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:58.257202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:18:58.257165 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:03.007432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:03.007395 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:07.732940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:07.732900 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:12.429500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:12.429456 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:16.005405 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:19:16.005370 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:19:17.206053 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:17.206018 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:17.417754 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:17.417716 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:21.904528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:21.904487 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:25.310365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:25.310324 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:26.719510 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:26.719480 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:31.724903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:31.724872 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:36.557677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:36.557622 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:37.391670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:37.391633 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r"] Nov 02 22:19:37.492008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:37.491977 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm"] Nov 02 22:19:37.862318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:37.862231 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:38.066478 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:38.066445 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:40.916929 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:40.916892 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:42.392837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.392769 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" podUID="c573ebd1-cb9d-4678-89ee-5afccf3b3954" containerName="payload-processing" containerID="cri-o://8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7" gracePeriod=25 Nov 02 22:19:42.402764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.402740 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:42.492621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.492588 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerName="payload-pre-processing" containerID="cri-o://e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51" gracePeriod=25 Nov 02 22:19:42.657819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.657759 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:19:42.661771 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.661742 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:42.707415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.706085 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb\" (UniqueName: \"kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb\") pod \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " Nov 02 22:19:42.707415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.706320 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume\") pod \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\" (UID: \"c573ebd1-cb9d-4678-89ee-5afccf3b3954\") " Nov 02 22:19:42.716806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.716683 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume" pod "c573ebd1-cb9d-4678-89ee-5afccf3b3954" (UID: "c573ebd1-cb9d-4678-89ee-5afccf3b3954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:42.719402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.719369 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb" pod "c573ebd1-cb9d-4678-89ee-5afccf3b3954" (UID: "c573ebd1-cb9d-4678-89ee-5afccf3b3954"). InnerVolumeSpecName "kube-api-access-bf8hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:42.807748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.807719 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c573ebd1-cb9d-4678-89ee-5afccf3b3954-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.807887 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.807753 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8hb\" (UniqueName: \"kubernetes.io/projected/c573ebd1-cb9d-4678-89ee-5afccf3b3954-kube-api-access-bf8hb\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.842951 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.842919 4285 generic.go:356] "Generic (PLEG): container finished" podID="c573ebd1-cb9d-4678-89ee-5afccf3b3954" containerID="8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7" exitCode=0 Nov 02 22:19:42.843115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.842983 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" event={"ID":"c573ebd1-cb9d-4678-89ee-5afccf3b3954","Type":"ContainerDied","Data":"8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7"} Nov 02 22:19:42.843115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.843020 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" event={"ID":"c573ebd1-cb9d-4678-89ee-5afccf3b3954","Type":"ContainerDied","Data":"5910eb8fe806f1382183aa446cacabf4ae8abc6c2141143c61f83e7cbeb22e54"} Nov 02 22:19:42.843115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.843044 4285 scope.go:122] "RemoveContainer" containerID="8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7" Nov 02 22:19:42.843324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.843196 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r" Nov 02 22:19:42.854525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.854503 4285 scope.go:122] "RemoveContainer" containerID="8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7" Nov 02 22:19:42.854816 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:19:42.854790 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7\": container with ID starting with 8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7 not found: ID does not exist" containerID="8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7" Nov 02 22:19:42.854889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.854828 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7"} err="failed to get container status \"8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7\": rpc error: code = NotFound desc = could not find container \"8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7\": container with ID starting with 8ba5bc196b05eadc0a984ab7c81c3844f0d5954a40d141f6e6ba6d9f9676c5c7 not found: ID does not exist" Nov 02 22:19:42.870504 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:42.870481 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:43.014935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:43.014912 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r"] Nov 02 22:19:43.017121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:43.017100 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1789c6-6dfbbc8dcc-lv26r"] Nov 02 22:19:43.172974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:43.172944 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c573ebd1-cb9d-4678-89ee-5afccf3b3954" path="/var/lib/kubelet/pods/c573ebd1-cb9d-4678-89ee-5afccf3b3954/volumes" Nov 02 22:19:43.459398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:43.459371 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:46.004360 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:19:46.004325 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:19:47.736418 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:47.736386 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:49.630190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:49.630160 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:49.878798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:49.878760 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:50.144008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:50.143971 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:50.365196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:50.365160 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:51.406188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:51.406120 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:19:52.828258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:52.828218 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:57.912283 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:19:57.912248 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:01.406240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:01.406203 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:20:02.644751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.644716 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:02.775574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.775550 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:20:02.813302 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.813273 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:02.913932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.913902 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume\") pod \"a4f54f65-697b-43af-abbe-f1e530906d7d\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " Nov 02 22:20:02.914174 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.914069 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh\" (UniqueName: \"kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh\") pod \"a4f54f65-697b-43af-abbe-f1e530906d7d\" (UID: \"a4f54f65-697b-43af-abbe-f1e530906d7d\") " Nov 02 22:20:02.914384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.914362 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume" pod "a4f54f65-697b-43af-abbe-f1e530906d7d" (UID: "a4f54f65-697b-43af-abbe-f1e530906d7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:02.914555 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.914532 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f54f65-697b-43af-abbe-f1e530906d7d-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:02.916670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.916639 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh" pod "a4f54f65-697b-43af-abbe-f1e530906d7d" (UID: "a4f54f65-697b-43af-abbe-f1e530906d7d"). InnerVolumeSpecName "kube-api-access-qm5nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:02.963328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.963304 4285 generic.go:356] "Generic (PLEG): container finished" podID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerID="e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51" exitCode=0 Nov 02 22:20:02.963446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.963358 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" Nov 02 22:20:02.963446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.963391 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" event={"ID":"a4f54f65-697b-43af-abbe-f1e530906d7d","Type":"ContainerDied","Data":"e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51"} Nov 02 22:20:02.963530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.963454 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm" event={"ID":"a4f54f65-697b-43af-abbe-f1e530906d7d","Type":"ContainerDied","Data":"27a17fa1f1e6fc2dd846c5957db89c1f80a483b1bbe3864703790802f3ffb066"} Nov 02 22:20:02.963530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.963478 4285 scope.go:122] "RemoveContainer" containerID="e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51" Nov 02 22:20:02.973175 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.973158 4285 scope.go:122] "RemoveContainer" containerID="e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51" Nov 02 22:20:02.973444 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:20:02.973427 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51\": container with ID starting with e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51 not found: ID does not exist" containerID="e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51" Nov 02 22:20:02.973490 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:02.973455 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51"} err="failed to get container status \"e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51\": rpc error: code = NotFound desc = could not find container \"e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51\": container with ID starting with e49039a3c2aceef821f7326449c29f9de44c64ee8de3a3818835b8b5f9b26d51 not found: ID does not exist" Nov 02 22:20:03.006209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:03.004946 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:03.016022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:03.015983 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm5nh\" (UniqueName: \"kubernetes.io/projected/a4f54f65-697b-43af-abbe-f1e530906d7d-kube-api-access-qm5nh\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:03.131218 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:03.131181 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm"] Nov 02 22:20:03.134916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:03.134885 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1789c6-5d7b76f8c5-tkftm"] Nov 02 22:20:03.172996 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:03.172967 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" path="/var/lib/kubelet/pods/a4f54f65-697b-43af-abbe-f1e530906d7d/volumes" Nov 02 22:20:07.469166 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:07.468874 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:12.417992 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:12.417952 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:16.006753 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:20:16.006705 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:20:17.378109 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:17.378073 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:17.780613 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:17.780573 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:18.385662 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:18.385629 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs"] Nov 02 22:20:18.386165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:18.385822 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" containerID="cri-o://7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92" gracePeriod=30 Nov 02 22:20:19.047307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:19.047275 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g"] Nov 02 22:20:19.097191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:19.097158 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-628d3872-zmg4g"] Nov 02 22:20:19.172878 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:19.172847 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f7ce76-401f-4e76-8f98-3c6d830201e3" path="/var/lib/kubelet/pods/27f7ce76-401f-4e76-8f98-3c6d830201e3/volumes" Nov 02 22:20:21.571909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.571864 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4"] Nov 02 22:20:21.572350 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.572078 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" podUID="3443b75b-8d6e-4388-828c-51520e369633" containerName="maas-api" containerID="cri-o://eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12" gracePeriod=30 Nov 02 22:20:21.700908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.700867 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:21.856710 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.856685 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:20:21.937858 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.937829 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\") pod \"3443b75b-8d6e-4388-828c-51520e369633\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " Nov 02 22:20:21.938009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.937960 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs\") pod \"3443b75b-8d6e-4388-828c-51520e369633\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " Nov 02 22:20:21.938009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.937987 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca\") pod \"3443b75b-8d6e-4388-828c-51520e369633\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " Nov 02 22:20:21.938109 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.938076 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9\" (UniqueName: \"kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9\") pod \"3443b75b-8d6e-4388-828c-51520e369633\" (UID: \"3443b75b-8d6e-4388-828c-51520e369633\") " Nov 02 22:20:21.938432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.938406 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca" pod "3443b75b-8d6e-4388-828c-51520e369633" (UID: "3443b75b-8d6e-4388-828c-51520e369633"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:21.940257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.940228 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls" pod "3443b75b-8d6e-4388-828c-51520e369633" (UID: "3443b75b-8d6e-4388-828c-51520e369633"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:21.940349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.940287 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9" pod "3443b75b-8d6e-4388-828c-51520e369633" (UID: "3443b75b-8d6e-4388-828c-51520e369633"). InnerVolumeSpecName "kube-api-access-f6lg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:21.940475 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:21.940456 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs" pod "3443b75b-8d6e-4388-828c-51520e369633" (UID: "3443b75b-8d6e-4388-828c-51520e369633"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:20:22.039473 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.039436 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.039473 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.039476 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3443b75b-8d6e-4388-828c-51520e369633-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.039677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.039490 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6lg9\" (UniqueName: \"kubernetes.io/projected/3443b75b-8d6e-4388-828c-51520e369633-kube-api-access-f6lg9\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.039677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.039505 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3443b75b-8d6e-4388-828c-51520e369633-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.069227 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.069191 4285 generic.go:356] "Generic (PLEG): container finished" podID="3443b75b-8d6e-4388-828c-51520e369633" containerID="eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12" exitCode=0 Nov 02 22:20:22.069412 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.069232 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" event={"ID":"3443b75b-8d6e-4388-828c-51520e369633","Type":"ContainerDied","Data":"eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12"} Nov 02 22:20:22.069412 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.069243 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" Nov 02 22:20:22.069412 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.069277 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4" event={"ID":"3443b75b-8d6e-4388-828c-51520e369633","Type":"ContainerDied","Data":"e9280b9f5f6fd0d48f3fecd13966ad1275ba44ace4077b299753cd3b84bba231"} Nov 02 22:20:22.069412 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.069299 4285 scope.go:122] "RemoveContainer" containerID="eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12" Nov 02 22:20:22.078986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.078969 4285 scope.go:122] "RemoveContainer" containerID="eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12" Nov 02 22:20:22.079233 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:20:22.079212 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12\": container with ID starting with eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12 not found: ID does not exist" containerID="eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12" Nov 02 22:20:22.079317 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.079240 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12"} err="failed to get container status \"eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12\": rpc error: code = NotFound desc = could not find container \"eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12\": container with ID starting with eec3d034b6b358abae5daaf89ee01dbaad9e1500c580bfc608469b092e85be12 not found: ID does not exist" Nov 02 22:20:22.243739 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.243699 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4"] Nov 02 22:20:22.248856 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.248828 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af9e7c-58b46fcdcb-f9zv4"] Nov 02 22:20:22.811420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:22.811390 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:23.172325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.172253 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3443b75b-8d6e-4388-828c-51520e369633" path="/var/lib/kubelet/pods/3443b75b-8d6e-4388-828c-51520e369633/volumes" Nov 02 22:20:23.284219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.284183 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:23.511451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.511413 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:23.643321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.643293 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:20:23.692505 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.692474 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:23.757791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757731 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.757791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757778 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.757984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757814 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.757984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757840 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.757984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757877 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.757984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757907 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.757984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757927 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.758286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.757982 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.758286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758094 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.758286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758106 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.758286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758120 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.758286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758157 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs\") pod \"5e889220-94d6-4ee1-a77b-d0fed0c099be\" (UID: \"5e889220-94d6-4ee1-a77b-d0fed0c099be\") " Nov 02 22:20:23.758567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758362 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.758567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758405 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:23.758676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758663 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.758741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758686 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.758741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758701 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.758741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758716 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.758741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.758730 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e889220-94d6-4ee1-a77b-d0fed0c099be-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.760065 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.759947 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:23.760065 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.759951 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:23.760065 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.760030 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "kube-api-access-8v7rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:23.760065 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.760040 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token" pod "5e889220-94d6-4ee1-a77b-d0fed0c099be" (UID: "5e889220-94d6-4ee1-a77b-d0fed0c099be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:23.859958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.859922 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.859958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.859960 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.860391 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.859975 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e889220-94d6-4ee1-a77b-d0fed0c099be-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:23.860391 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:23.859989 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v7rx\" (UniqueName: \"kubernetes.io/projected/5e889220-94d6-4ee1-a77b-d0fed0c099be-kube-api-access-8v7rx\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:24.083613 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.083526 4285 generic.go:356] "Generic (PLEG): container finished" podID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerID="7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92" exitCode=0 Nov 02 22:20:24.083613 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.083581 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" Nov 02 22:20:24.083613 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.083589 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" event={"ID":"5e889220-94d6-4ee1-a77b-d0fed0c099be","Type":"ContainerDied","Data":"7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92"} Nov 02 22:20:24.083902 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.083669 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs" event={"ID":"5e889220-94d6-4ee1-a77b-d0fed0c099be","Type":"ContainerDied","Data":"3c1b6e5acadeec40112b45c82d3767b653ced528914be8ecf7064a4cead4b8f5"} Nov 02 22:20:24.083902 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.083700 4285 scope.go:122] "RemoveContainer" containerID="7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92" Nov 02 22:20:24.096497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.096474 4285 scope.go:122] "RemoveContainer" containerID="7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92" Nov 02 22:20:24.096899 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:20:24.096863 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92\": container with ID starting with 7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92 not found: ID does not exist" containerID="7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92" Nov 02 22:20:24.096985 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.096911 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92"} err="failed to get container status \"7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92\": rpc error: code = NotFound desc = could not find container \"7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92\": container with ID starting with 7a827a45488a3c5a2f0a4be89d9de55e5f4a7d26e2c8e0117be4615de6b4ac92 not found: ID does not exist" Nov 02 22:20:24.236293 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.236255 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:24.248894 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.248867 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs"] Nov 02 22:20:24.298533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:24.298502 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-5zlhs"] Nov 02 22:20:25.173522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:25.173490 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" path="/var/lib/kubelet/pods/5e889220-94d6-4ee1-a77b-d0fed0c099be/volumes" Nov 02 22:20:28.585898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:28.585861 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:31.848911 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:31.848875 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:32.111085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:32.110989 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:32.348687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:32.348649 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:33.820481 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:33.820450 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:39.020910 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:39.020869 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:43.387399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:43.387358 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:43.817136 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:43.817093 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:44.960745 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:44.960716 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj"] Nov 02 22:20:45.074182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:45.074151 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f"] Nov 02 22:20:45.342653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:45.342577 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:45.523911 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:45.523875 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:46.004636 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:20:46.004604 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:20:48.611648 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:48.611616 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:49.961247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:49.961202 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" podUID="e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" containerName="payload-processing" containerID="cri-o://314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222" gracePeriod=25 Nov 02 22:20:50.075022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.074961 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" podUID="7058b902-fcd3-411b-af6b-d4d63b8a4cba" containerName="payload-pre-processing" containerID="cri-o://61ce2074b27e48297c8d4d8fb940083dbdadc9f326d6822526ffb18300f7685a" gracePeriod=25 Nov 02 22:20:50.231000 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.230932 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:20:50.234488 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.234462 4285 generic.go:356] "Generic (PLEG): container finished" podID="e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" containerID="314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222" exitCode=0 Nov 02 22:20:50.234672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.234515 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" Nov 02 22:20:50.234672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.234547 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" event={"ID":"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87","Type":"ContainerDied","Data":"314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222"} Nov 02 22:20:50.234672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.234608 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj" event={"ID":"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87","Type":"ContainerDied","Data":"98bd75f299a68b2eabaae53a95c6f1d2ffa86fe27abf4596b938f159c8bef2c9"} Nov 02 22:20:50.234672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.234641 4285 scope.go:122] "RemoveContainer" containerID="314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222" Nov 02 22:20:50.237076 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.237053 4285 generic.go:356] "Generic (PLEG): container finished" podID="7058b902-fcd3-411b-af6b-d4d63b8a4cba" containerID="61ce2074b27e48297c8d4d8fb940083dbdadc9f326d6822526ffb18300f7685a" exitCode=0 Nov 02 22:20:50.237444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.237221 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" event={"ID":"7058b902-fcd3-411b-af6b-d4d63b8a4cba","Type":"ContainerDied","Data":"61ce2074b27e48297c8d4d8fb940083dbdadc9f326d6822526ffb18300f7685a"} Nov 02 22:20:50.248230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.244875 4285 scope.go:122] "RemoveContainer" containerID="314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222" Nov 02 22:20:50.248230 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:20:50.245397 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222\": container with ID starting with 314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222 not found: ID does not exist" containerID="314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222" Nov 02 22:20:50.248230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.245430 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222"} err="failed to get container status \"314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222\": rpc error: code = NotFound desc = could not find container \"314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222\": container with ID starting with 314cdd44a5602ced79c9c399e957929928ef4eec305b52569f9d7e9095fd7222 not found: ID does not exist" Nov 02 22:20:50.265634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.265608 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume\") pod \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " Nov 02 22:20:50.265768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.265751 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn\" (UniqueName: \"kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn\") pod \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\" (UID: \"e5a92a95-a4bf-4805-a8ce-e2b0c0206c87\") " Nov 02 22:20:50.266028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.266001 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume" pod "e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" (UID: "e5a92a95-a4bf-4805-a8ce-e2b0c0206c87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:50.266360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.266343 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.267857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.267831 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn" pod "e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" (UID: "e5a92a95-a4bf-4805-a8ce-e2b0c0206c87"). InnerVolumeSpecName "kube-api-access-b4pvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:50.319444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.319421 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:20:50.367518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.367494 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume\") pod \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " Nov 02 22:20:50.367698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.367535 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p\" (UniqueName: \"kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p\") pod \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\" (UID: \"7058b902-fcd3-411b-af6b-d4d63b8a4cba\") " Nov 02 22:20:50.367809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.367795 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pvn\" (UniqueName: \"kubernetes.io/projected/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87-kube-api-access-b4pvn\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.367844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.367801 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume" pod "7058b902-fcd3-411b-af6b-d4d63b8a4cba" (UID: "7058b902-fcd3-411b-af6b-d4d63b8a4cba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:50.369637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.369604 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p" pod "7058b902-fcd3-411b-af6b-d4d63b8a4cba" (UID: "7058b902-fcd3-411b-af6b-d4d63b8a4cba"). InnerVolumeSpecName "kube-api-access-h9n4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:50.469298 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.469264 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058b902-fcd3-411b-af6b-d4d63b8a4cba-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.469298 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.469298 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9n4p\" (UniqueName: \"kubernetes.io/projected/7058b902-fcd3-411b-af6b-d4d63b8a4cba-kube-api-access-h9n4p\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:20:50.653353 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.653313 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj"] Nov 02 22:20:50.656972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:50.656946 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fb0367-7bb6669fc7-b5fjj"] Nov 02 22:20:51.172770 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:51.172738 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" path="/var/lib/kubelet/pods/e5a92a95-a4bf-4805-a8ce-e2b0c0206c87/volumes" Nov 02 22:20:51.245099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:51.245039 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" Nov 02 22:20:51.245271 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:51.245038 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f" event={"ID":"7058b902-fcd3-411b-af6b-d4d63b8a4cba","Type":"ContainerDied","Data":"e30d478665df53c9fa3719a8a87a09bbbe2ca52e5a87959ec919bd83702505a2"} Nov 02 22:20:51.245271 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:51.245190 4285 scope.go:122] "RemoveContainer" containerID="61ce2074b27e48297c8d4d8fb940083dbdadc9f326d6822526ffb18300f7685a" Nov 02 22:20:51.410254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:51.410226 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f"] Nov 02 22:20:51.460370 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:51.460329 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fb0367-7bc94bc6c5-6fw9f"] Nov 02 22:20:53.173296 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:53.173266 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7058b902-fcd3-411b-af6b-d4d63b8a4cba" path="/var/lib/kubelet/pods/7058b902-fcd3-411b-af6b-d4d63b8a4cba/volumes" Nov 02 22:20:53.410449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:53.410404 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:58.266298 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:20:58.266256 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:02.924011 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:02.923976 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:06.888320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:06.888285 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg"] Nov 02 22:21:06.888709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:06.888529 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" podUID="8a24076f-b642-4d94-bb34-5211faf082ff" containerName="maas-api" containerID="cri-o://54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a" gracePeriod=30 Nov 02 22:21:07.021344 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.021303 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:07.157796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.157771 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:21:07.237809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.237779 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\") pod \"8a24076f-b642-4d94-bb34-5211faf082ff\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " Nov 02 22:21:07.237948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.237821 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs\") pod \"8a24076f-b642-4d94-bb34-5211faf082ff\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " Nov 02 22:21:07.237948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.237938 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca\") pod \"8a24076f-b642-4d94-bb34-5211faf082ff\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " Nov 02 22:21:07.238073 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.238014 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz\" (UniqueName: \"kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz\") pod \"8a24076f-b642-4d94-bb34-5211faf082ff\" (UID: \"8a24076f-b642-4d94-bb34-5211faf082ff\") " Nov 02 22:21:07.238324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.238296 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca" pod "8a24076f-b642-4d94-bb34-5211faf082ff" (UID: "8a24076f-b642-4d94-bb34-5211faf082ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:07.238484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.238464 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a24076f-b642-4d94-bb34-5211faf082ff-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:07.239811 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.239786 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls" pod "8a24076f-b642-4d94-bb34-5211faf082ff" (UID: "8a24076f-b642-4d94-bb34-5211faf082ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:07.239908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.239870 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz" pod "8a24076f-b642-4d94-bb34-5211faf082ff" (UID: "8a24076f-b642-4d94-bb34-5211faf082ff"). InnerVolumeSpecName "kube-api-access-p48tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:07.239968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.239927 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs" pod "8a24076f-b642-4d94-bb34-5211faf082ff" (UID: "8a24076f-b642-4d94-bb34-5211faf082ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:07.328803 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.328770 4285 generic.go:356] "Generic (PLEG): container finished" podID="8a24076f-b642-4d94-bb34-5211faf082ff" containerID="54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a" exitCode=0 Nov 02 22:21:07.328954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.328820 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" Nov 02 22:21:07.328954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.328848 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" event={"ID":"8a24076f-b642-4d94-bb34-5211faf082ff","Type":"ContainerDied","Data":"54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a"} Nov 02 22:21:07.328954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.328904 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg" event={"ID":"8a24076f-b642-4d94-bb34-5211faf082ff","Type":"ContainerDied","Data":"30a086054a29285c6467c0b96ae841cb0b14d2459cc63cdd191af3dbfdab1b69"} Nov 02 22:21:07.328954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.328929 4285 scope.go:122] "RemoveContainer" containerID="54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a" Nov 02 22:21:07.340390 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.340365 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p48tz\" (UniqueName: \"kubernetes.io/projected/8a24076f-b642-4d94-bb34-5211faf082ff-kube-api-access-p48tz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:07.340390 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.340396 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:07.340534 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.340414 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a24076f-b642-4d94-bb34-5211faf082ff-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:07.341184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.341168 4285 scope.go:122] "RemoveContainer" containerID="54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a" Nov 02 22:21:07.341475 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:21:07.341455 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a\": container with ID starting with 54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a not found: ID does not exist" containerID="54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a" Nov 02 22:21:07.341540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.341481 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a"} err="failed to get container status \"54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a\": rpc error: code = NotFound desc = could not find container \"54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a\": container with ID starting with 54b830834988c1846ac00f99988dafd05a587d3e98c2dd81329761277d78208a not found: ID does not exist" Nov 02 22:21:07.491833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.491799 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg"] Nov 02 22:21:07.495870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:07.495838 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-fb0367-c47957c4-bpjtg"] Nov 02 22:21:08.514537 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:08.514469 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:08.847769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:08.847633 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:09.031549 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:09.031512 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:09.172956 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:09.172883 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a24076f-b642-4d94-bb34-5211faf082ff" path="/var/lib/kubelet/pods/8a24076f-b642-4d94-bb34-5211faf082ff/volumes" Nov 02 22:21:09.457296 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:09.457241 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:13.704577 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:13.704538 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:15.348667 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:15.348635 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:15.507956 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:15.507918 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:15.670861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:15.670787 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:16.004675 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:21:16.004646 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:21:18.084880 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:18.084843 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:18.665410 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:18.665380 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:19.341047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:19.341013 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:23.675409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:23.675377 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:24.350562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:24.350532 4285 scope.go:122] "RemoveContainer" containerID="aaa4c06d0d5fddd8ac6932616cf957cf563d3505b641a1e116d16417aa51f91b" Nov 02 22:21:28.103309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:28.103277 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb"] Nov 02 22:21:28.387669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:28.387592 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:28.538684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:28.538644 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:28.850260 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:28.850223 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:32.962495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:32.962461 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67"] Nov 02 22:21:32.962978 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:32.962651 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" podUID="d2804ab9-431b-45f3-b8ef-23f7c27b2c51" containerName="maas-api" containerID="cri-o://2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562" gracePeriod=30 Nov 02 22:21:33.083439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.083413 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:33.103653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.103620 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" podUID="b1c42c48-894a-478b-858e-304f21ebcbff" containerName="payload-pre-processing" containerID="cri-o://c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666" gracePeriod=25 Nov 02 22:21:33.227021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.227001 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:21:33.305733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.305702 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs\") pod \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " Nov 02 22:21:33.305890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.305757 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls\") pod \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " Nov 02 22:21:33.305890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.305789 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv\" (UniqueName: \"kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv\") pod \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " Nov 02 22:21:33.305890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.305871 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca\") pod \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\" (UID: \"d2804ab9-431b-45f3-b8ef-23f7c27b2c51\") " Nov 02 22:21:33.306467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.306436 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca" pod "d2804ab9-431b-45f3-b8ef-23f7c27b2c51" (UID: "d2804ab9-431b-45f3-b8ef-23f7c27b2c51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:33.307933 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.307909 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls" pod "d2804ab9-431b-45f3-b8ef-23f7c27b2c51" (UID: "d2804ab9-431b-45f3-b8ef-23f7c27b2c51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:33.308059 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.307944 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv" pod "d2804ab9-431b-45f3-b8ef-23f7c27b2c51" (UID: "d2804ab9-431b-45f3-b8ef-23f7c27b2c51"). InnerVolumeSpecName "kube-api-access-4hkbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:33.308430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.308412 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs" pod "d2804ab9-431b-45f3-b8ef-23f7c27b2c51" (UID: "d2804ab9-431b-45f3-b8ef-23f7c27b2c51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:33.377005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.376974 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:21:33.406725 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.406693 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm\" (UniqueName: \"kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm\") pod \"b1c42c48-894a-478b-858e-304f21ebcbff\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " Nov 02 22:21:33.406860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.406835 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume\") pod \"b1c42c48-894a-478b-858e-304f21ebcbff\" (UID: \"b1c42c48-894a-478b-858e-304f21ebcbff\") " Nov 02 22:21:33.407222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.407198 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume" pod "b1c42c48-894a-478b-858e-304f21ebcbff" (UID: "b1c42c48-894a-478b-858e-304f21ebcbff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:33.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.407227 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.407246 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.407255 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.407279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.407264 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkbv\" (UniqueName: \"kubernetes.io/projected/d2804ab9-431b-45f3-b8ef-23f7c27b2c51-kube-api-access-4hkbv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.408859 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.408830 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm" pod "b1c42c48-894a-478b-858e-304f21ebcbff" (UID: "b1c42c48-894a-478b-858e-304f21ebcbff"). InnerVolumeSpecName "kube-api-access-sczkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:33.477797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.477725 4285 generic.go:356] "Generic (PLEG): container finished" podID="b1c42c48-894a-478b-858e-304f21ebcbff" containerID="c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666" exitCode=0 Nov 02 22:21:33.477797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.477771 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" Nov 02 22:21:33.477994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.477809 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" event={"ID":"b1c42c48-894a-478b-858e-304f21ebcbff","Type":"ContainerDied","Data":"c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666"} Nov 02 22:21:33.477994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.477859 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb" event={"ID":"b1c42c48-894a-478b-858e-304f21ebcbff","Type":"ContainerDied","Data":"ba01a55b3665a4201d4e363c3aa487585ca195dabd9c1eb7381f9a79b97b659f"} Nov 02 22:21:33.477994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.477889 4285 scope.go:122] "RemoveContainer" containerID="c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666" Nov 02 22:21:33.484557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.480597 4285 generic.go:356] "Generic (PLEG): container finished" podID="d2804ab9-431b-45f3-b8ef-23f7c27b2c51" containerID="2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562" exitCode=0 Nov 02 22:21:33.484557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.480677 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" event={"ID":"d2804ab9-431b-45f3-b8ef-23f7c27b2c51","Type":"ContainerDied","Data":"2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562"} Nov 02 22:21:33.484557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.480716 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" event={"ID":"d2804ab9-431b-45f3-b8ef-23f7c27b2c51","Type":"ContainerDied","Data":"1a38e35159cd1fdeace74d9caf5b7a5125ec21fbfe89eeec503fa3140ee2e927"} Nov 02 22:21:33.484557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.480779 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67" Nov 02 22:21:33.494037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.494018 4285 scope.go:122] "RemoveContainer" containerID="c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666" Nov 02 22:21:33.494381 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:21:33.494360 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666\": container with ID starting with c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666 not found: ID does not exist" containerID="c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666" Nov 02 22:21:33.494461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.494389 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666"} err="failed to get container status \"c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666\": rpc error: code = NotFound desc = could not find container \"c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666\": container with ID starting with c328d742ffc29f8f5f3f28aa9965300bd8a1ada34b4f58d97810c50b9b2cb666 not found: ID does not exist" Nov 02 22:21:33.494461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.494410 4285 scope.go:122] "RemoveContainer" containerID="2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562" Nov 02 22:21:33.507479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.507457 4285 scope.go:122] "RemoveContainer" containerID="2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562" Nov 02 22:21:33.507760 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:21:33.507738 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562\": container with ID starting with 2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562 not found: ID does not exist" containerID="2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562" Nov 02 22:21:33.507875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.507771 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562"} err="failed to get container status \"2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562\": rpc error: code = NotFound desc = could not find container \"2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562\": container with ID starting with 2e5b0fb2ea7b2c9862c91725b3adc25ad337406f351136dfb61cca7f08b5a562 not found: ID does not exist" Nov 02 22:21:33.508045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.508024 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczkm\" (UniqueName: \"kubernetes.io/projected/b1c42c48-894a-478b-858e-304f21ebcbff-kube-api-access-sczkm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.508118 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.508048 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c42c48-894a-478b-858e-304f21ebcbff-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.586538 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.586466 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:33.599468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.599439 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb"] Nov 02 22:21:33.602649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.602625 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-267d89-579b446867-lf6tb"] Nov 02 22:21:33.739965 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.739891 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:33.763088 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.763061 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67"] Nov 02 22:21:33.766187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:33.766163 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d1a155-7b6ddccd59-92l67"] Nov 02 22:21:34.841916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:34.841880 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:35.173507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:35.173434 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c42c48-894a-478b-858e-304f21ebcbff" path="/var/lib/kubelet/pods/b1c42c48-894a-478b-858e-304f21ebcbff/volumes" Nov 02 22:21:35.173848 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:35.173835 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2804ab9-431b-45f3-b8ef-23f7c27b2c51" path="/var/lib/kubelet/pods/d2804ab9-431b-45f3-b8ef-23f7c27b2c51/volumes" Nov 02 22:21:37.284627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:37.284591 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:37.563321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:37.563240 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:38.138968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:38.138933 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:39.818229 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:39.818193 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:43.831756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:43.831721 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:43.948279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:43.948241 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x"] Nov 02 22:21:43.948492 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:43.948467 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" podUID="eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" containerName="maas-api" containerID="cri-o://32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a" gracePeriod=30 Nov 02 22:21:43.991315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:43.991272 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:44.163575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.163542 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:44.238110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.238083 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:21:44.330148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.325874 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs\") pod \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " Nov 02 22:21:44.330148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.325941 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn\" (UniqueName: \"kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn\") pod \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " Nov 02 22:21:44.330148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.325974 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\") pod \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " Nov 02 22:21:44.330148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.326001 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca\") pod \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\" (UID: \"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869\") " Nov 02 22:21:44.330148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.326226 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:44.330148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.328045 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca" pod "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" (UID: "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:44.331809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.331756 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs" pod "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" (UID: "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:44.332252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.332219 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn" pod "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" (UID: "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869"). InnerVolumeSpecName "kube-api-access-6wxtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:44.332356 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.332296 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls" pod "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" (UID: "eb47c9e0-ddf8-4953-ade9-1d2bfeeec869"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:44.427715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.427635 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:44.427715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.427660 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxtn\" (UniqueName: \"kubernetes.io/projected/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-kube-api-access-6wxtn\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:44.427715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.427672 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:44.427715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.427680 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:21:44.544111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.544075 4285 generic.go:356] "Generic (PLEG): container finished" podID="eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" containerID="32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a" exitCode=0 Nov 02 22:21:44.544305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.544106 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" event={"ID":"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869","Type":"ContainerDied","Data":"32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a"} Nov 02 22:21:44.544305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.544117 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" Nov 02 22:21:44.544305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.544216 4285 scope.go:122] "RemoveContainer" containerID="32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a" Nov 02 22:21:44.544305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.544202 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x" event={"ID":"eb47c9e0-ddf8-4953-ade9-1d2bfeeec869","Type":"ContainerDied","Data":"5b92fed6222f1265cecd594c8bc42e45c68abf7e2cf6d5f6b82633e4ef5e378a"} Nov 02 22:21:44.555169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.555151 4285 scope.go:122] "RemoveContainer" containerID="32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a" Nov 02 22:21:44.555452 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:21:44.555428 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a\": container with ID starting with 32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a not found: ID does not exist" containerID="32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a" Nov 02 22:21:44.555511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.555465 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a"} err="failed to get container status \"32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a\": rpc error: code = NotFound desc = could not find container \"32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a\": container with ID starting with 32ab7e3c3b8b2d3eeb571d99cbc8c319e54a458abb4b3a7ad6673ed0c8d3ae1a not found: ID does not exist" Nov 02 22:21:44.707635 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.707602 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x"] Nov 02 22:21:44.710363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:44.710339 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-267d89-7db6c894b5-bqk2x"] Nov 02 22:21:45.173682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:45.173608 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" path="/var/lib/kubelet/pods/eb47c9e0-ddf8-4953-ade9-1d2bfeeec869/volumes" Nov 02 22:21:45.674152 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:45.674083 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:46.003967 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:21:46.003935 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:21:46.014514 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:46.014471 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:46.167672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:46.167640 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:46.670436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:46.670391 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:50.993836 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:50.993801 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:52.381765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:52.381690 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:52.558161 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:52.558095 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:52.722429 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:52.722396 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:53.157014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:53.156904 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:56.086057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:56.086024 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:56.354175 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:56.354098 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk"] Nov 02 22:21:56.458093 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:56.458065 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd"] Nov 02 22:21:56.736402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:56.736368 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:56.892896 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:56.892860 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:58.103305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:58.103217 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6"] Nov 02 22:21:58.134862 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:58.134831 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk"] Nov 02 22:21:58.142872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:58.142838 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk"] Nov 02 22:21:58.144837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:58.144813 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7"] Nov 02 22:21:58.218806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:21:58.218778 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:00.396133 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:00.396080 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:01.354807 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.354752 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" podUID="33952884-fe06-4862-9077-7cfbc526fa76" containerName="payload-processing" containerID="cri-o://d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697" gracePeriod=25 Nov 02 22:22:01.458651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.458602 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerName="payload-pre-processing" containerID="cri-o://15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c" gracePeriod=25 Nov 02 22:22:01.602971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.602938 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:22:01.636330 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.636269 4285 generic.go:356] "Generic (PLEG): container finished" podID="33952884-fe06-4862-9077-7cfbc526fa76" containerID="d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697" exitCode=0 Nov 02 22:22:01.636330 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.636313 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" Nov 02 22:22:01.636466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.636349 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" event={"ID":"33952884-fe06-4862-9077-7cfbc526fa76","Type":"ContainerDied","Data":"d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697"} Nov 02 22:22:01.636466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.636421 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk" event={"ID":"33952884-fe06-4862-9077-7cfbc526fa76","Type":"ContainerDied","Data":"3d7bb997ee4029d1db543cad086ba49b223452031d763ff6b8e539d62762fc62"} Nov 02 22:22:01.636466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.636443 4285 scope.go:122] "RemoveContainer" containerID="d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697" Nov 02 22:22:01.644532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.644514 4285 scope.go:122] "RemoveContainer" containerID="d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697" Nov 02 22:22:01.644803 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:01.644775 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697\": container with ID starting with d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697 not found: ID does not exist" containerID="d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697" Nov 02 22:22:01.644875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.644807 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697"} err="failed to get container status \"d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697\": rpc error: code = NotFound desc = could not find container \"d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697\": container with ID starting with d8b39383dafaf8840438f87b5a07d955281012a2aa0e30a20cb904433e021697 not found: ID does not exist" Nov 02 22:22:01.703444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.703425 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx\" (UniqueName: \"kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx\") pod \"33952884-fe06-4862-9077-7cfbc526fa76\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " Nov 02 22:22:01.703566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.703489 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume\") pod \"33952884-fe06-4862-9077-7cfbc526fa76\" (UID: \"33952884-fe06-4862-9077-7cfbc526fa76\") " Nov 02 22:22:01.703883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.703860 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume" pod "33952884-fe06-4862-9077-7cfbc526fa76" (UID: "33952884-fe06-4862-9077-7cfbc526fa76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:01.705473 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.705445 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx" pod "33952884-fe06-4862-9077-7cfbc526fa76" (UID: "33952884-fe06-4862-9077-7cfbc526fa76"). InnerVolumeSpecName "kube-api-access-xvvzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:01.804809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.804778 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvzx\" (UniqueName: \"kubernetes.io/projected/33952884-fe06-4862-9077-7cfbc526fa76-kube-api-access-xvvzx\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:01.804809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.804810 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33952884-fe06-4862-9077-7cfbc526fa76-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:01.883003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:01.882956 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:02.055033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:02.054999 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk"] Nov 02 22:22:02.058096 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:02.058072 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-94a8a3-5f8cf6b459-lf9dk"] Nov 02 22:22:02.112194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:02.112135 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:02.260618 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:02.260529 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:02.767447 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:02.767409 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:03.172756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:03.172678 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33952884-fe06-4862-9077-7cfbc526fa76" path="/var/lib/kubelet/pods/33952884-fe06-4862-9077-7cfbc526fa76/volumes" Nov 02 22:22:05.427786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:05.427754 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:05.582240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:05.582205 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:07.206599 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:07.206568 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:07.427295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:07.427255 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:22:08.891348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:08.891314 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:09.043611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:09.043580 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:09.191184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:09.191085 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw"] Nov 02 22:22:09.191399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:09.191374 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" containerID="cri-o://2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962" gracePeriod=30 Nov 02 22:22:09.215552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:09.215520 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:09.675099 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:09.675060 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:12.489786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:12.489755 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:13.126230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.126181 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" containerID="cri-o://335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb" gracePeriod=45 Nov 02 22:22:13.154631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.154597 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" containerID="cri-o://af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f" gracePeriod=45 Nov 02 22:22:13.164957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.164924 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" containerID="cri-o://4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463" gracePeriod=45 Nov 02 22:22:13.166019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.165972 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" containerID="cri-o://660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260" gracePeriod=45 Nov 02 22:22:13.443329 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.440534 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 22:22:13.464092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.464063 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 22:22:13.468296 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.468275 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 22:22:13.473869 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.473851 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 22:22:13.507783 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507758 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache\") pod \"fceaaaee-d6ec-41a0-92cd-63dad84030af\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507790 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm\") pod \"fceaaaee-d6ec-41a0-92cd-63dad84030af\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507811 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache\") pod \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507829 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm\") pod \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507862 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg\" (UniqueName: \"kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg\") pod \"6784d5f2-659f-45cf-b603-99f19eae6473\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507884 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache\") pod \"6784d5f2-659f-45cf-b603-99f19eae6473\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507907 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home\") pod \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507939 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs\") pod \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.507983 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs\" (UniqueName: \"kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs\") pod \"6784d5f2-659f-45cf-b603-99f19eae6473\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508019 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs\") pod \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508073 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home\") pod \"6784d5f2-659f-45cf-b603-99f19eae6473\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " Nov 02 22:22:13.508149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508078 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache" pod "e6a13596-0a3e-45e4-ab6e-25c06f8abb13" (UID: "e6a13596-0a3e-45e4-ab6e-25c06f8abb13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508215 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir\") pod \"fceaaaee-d6ec-41a0-92cd-63dad84030af\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508254 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir\") pod \"6784d5f2-659f-45cf-b603-99f19eae6473\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508292 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir\") pod \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508297 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home" pod "6784d5f2-659f-45cf-b603-99f19eae6473" (UID: "6784d5f2-659f-45cf-b603-99f19eae6473"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508319 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm\") pod \"6784d5f2-659f-45cf-b603-99f19eae6473\" (UID: \"6784d5f2-659f-45cf-b603-99f19eae6473\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508341 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache" pod "6784d5f2-659f-45cf-b603-99f19eae6473" (UID: "6784d5f2-659f-45cf-b603-99f19eae6473"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508346 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache" pod "fceaaaee-d6ec-41a0-92cd-63dad84030af" (UID: "fceaaaee-d6ec-41a0-92cd-63dad84030af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508368 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home\") pod \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508453 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home\") pod \"fceaaaee-d6ec-41a0-92cd-63dad84030af\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508484 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm\") pod \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508511 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache\") pod \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508537 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir\") pod \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508543 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir" pod "6784d5f2-659f-45cf-b603-99f19eae6473" (UID: "6784d5f2-659f-45cf-b603-99f19eae6473"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508555 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home" pod "e6a13596-0a3e-45e4-ab6e-25c06f8abb13" (UID: "e6a13596-0a3e-45e4-ab6e-25c06f8abb13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508567 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs\" (UniqueName: \"kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs\") pod \"fceaaaee-d6ec-41a0-92cd-63dad84030af\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " Nov 02 22:22:13.508698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508635 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk\" (UniqueName: \"kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk\") pod \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\" (UID: \"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b\") " Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508719 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd\" (UniqueName: \"kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd\") pod \"fceaaaee-d6ec-41a0-92cd-63dad84030af\" (UID: \"fceaaaee-d6ec-41a0-92cd-63dad84030af\") " Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508729 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir" pod "fceaaaee-d6ec-41a0-92cd-63dad84030af" (UID: "fceaaaee-d6ec-41a0-92cd-63dad84030af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.508758 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m\" (UniqueName: \"kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m\") pod \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\" (UID: \"e6a13596-0a3e-45e4-ab6e-25c06f8abb13\") " Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509234 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509258 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509273 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509288 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509303 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509315 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.509502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.509327 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.510938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510002 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home" pod "fceaaaee-d6ec-41a0-92cd-63dad84030af" (UID: "fceaaaee-d6ec-41a0-92cd-63dad84030af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.510938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510193 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir" pod "e6a13596-0a3e-45e4-ab6e-25c06f8abb13" (UID: "e6a13596-0a3e-45e4-ab6e-25c06f8abb13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.510938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510383 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home" pod "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" (UID: "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.510938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510547 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache" pod "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" (UID: "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.510938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510722 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg" pod "6784d5f2-659f-45cf-b603-99f19eae6473" (UID: "6784d5f2-659f-45cf-b603-99f19eae6473"). InnerVolumeSpecName "kube-api-access-mrdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:13.510938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510783 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm" pod "fceaaaee-d6ec-41a0-92cd-63dad84030af" (UID: "fceaaaee-d6ec-41a0-92cd-63dad84030af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.511257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510969 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir" pod "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" (UID: "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.511257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.510986 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs" pod "e6a13596-0a3e-45e4-ab6e-25c06f8abb13" (UID: "e6a13596-0a3e-45e4-ab6e-25c06f8abb13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:13.511257 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.511071 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs" pod "fceaaaee-d6ec-41a0-92cd-63dad84030af" (UID: "fceaaaee-d6ec-41a0-92cd-63dad84030af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:13.512344 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.512320 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm" pod "e6a13596-0a3e-45e4-ab6e-25c06f8abb13" (UID: "e6a13596-0a3e-45e4-ab6e-25c06f8abb13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.512722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.512686 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs" pod "6784d5f2-659f-45cf-b603-99f19eae6473" (UID: "6784d5f2-659f-45cf-b603-99f19eae6473"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:13.512811 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.512783 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd" pod "fceaaaee-d6ec-41a0-92cd-63dad84030af" (UID: "fceaaaee-d6ec-41a0-92cd-63dad84030af"). InnerVolumeSpecName "kube-api-access-vh8zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:13.512887 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.512872 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm" pod "6784d5f2-659f-45cf-b603-99f19eae6473" (UID: "6784d5f2-659f-45cf-b603-99f19eae6473"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.512958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.512928 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs" pod "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" (UID: "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:13.513337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.513314 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk" pod "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" (UID: "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b"). InnerVolumeSpecName "kube-api-access-pthqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:13.513484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.513467 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m" pod "e6a13596-0a3e-45e4-ab6e-25c06f8abb13" (UID: "e6a13596-0a3e-45e4-ab6e-25c06f8abb13"). InnerVolumeSpecName "kube-api-access-xv62m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:13.513849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.513834 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm" pod "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" (UID: "1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:13.610154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610104 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6784d5f2-659f-45cf-b603-99f19eae6473-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610158 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610173 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610188 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610203 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610218 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610234 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fceaaaee-d6ec-41a0-92cd-63dad84030af-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610248 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthqk\" (UniqueName: \"kubernetes.io/projected/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-kube-api-access-pthqk\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610264 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8zd\" (UniqueName: \"kubernetes.io/projected/fceaaaee-d6ec-41a0-92cd-63dad84030af-kube-api-access-vh8zd\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610279 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv62m\" (UniqueName: \"kubernetes.io/projected/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-kube-api-access-xv62m\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610297 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fceaaaee-d6ec-41a0-92cd-63dad84030af-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610313 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610327 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrdtg\" (UniqueName: \"kubernetes.io/projected/6784d5f2-659f-45cf-b603-99f19eae6473-kube-api-access-mrdtg\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610341 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610356 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6784d5f2-659f-45cf-b603-99f19eae6473-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610373 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.610738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.610387 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a13596-0a3e-45e4-ab6e-25c06f8abb13-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.711613 4285 generic.go:356] "Generic (PLEG): container finished" podID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerID="660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260" exitCode=0 Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.711656 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" event={"ID":"fceaaaee-d6ec-41a0-92cd-63dad84030af","Type":"ContainerDied","Data":"660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260"} Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.711714 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" event={"ID":"fceaaaee-d6ec-41a0-92cd-63dad84030af","Type":"ContainerDied","Data":"fada2780ccb4b0569baeb952d335058bd06fe8a3cad046ad26fb508042c91700"} Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.711736 4285 scope.go:122] "RemoveContainer" containerID="660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260" Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.711674 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk" Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.713706 4285 generic.go:356] "Generic (PLEG): container finished" podID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerID="af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f" exitCode=0 Nov 02 22:22:13.717452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.713798 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" Nov 02 22:22:13.718720 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.718691 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" event={"ID":"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b","Type":"ContainerDied","Data":"af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f"} Nov 02 22:22:13.718845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.718747 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk" event={"ID":"1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b","Type":"ContainerDied","Data":"deece6b6c5c7281b1cf6fd59d3e736664e4a37e5ded2327cf2c19df397be872f"} Nov 02 22:22:13.723858 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.723203 4285 generic.go:356] "Generic (PLEG): container finished" podID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerID="335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb" exitCode=0 Nov 02 22:22:13.723858 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.723287 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" event={"ID":"e6a13596-0a3e-45e4-ab6e-25c06f8abb13","Type":"ContainerDied","Data":"335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb"} Nov 02 22:22:13.723858 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.723315 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" event={"ID":"e6a13596-0a3e-45e4-ab6e-25c06f8abb13","Type":"ContainerDied","Data":"c3d3fe2f28ac20d9cbbeaddb38ca5217bf1e367967c0fc7a136e7bffa6b52aaf"} Nov 02 22:22:13.723858 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.723378 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6" Nov 02 22:22:13.745472 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.745443 4285 scope.go:122] "RemoveContainer" containerID="660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260" Nov 02 22:22:13.745748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.745476 4285 generic.go:356] "Generic (PLEG): container finished" podID="6784d5f2-659f-45cf-b603-99f19eae6473" containerID="4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463" exitCode=0 Nov 02 22:22:13.745897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.745504 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" event={"ID":"6784d5f2-659f-45cf-b603-99f19eae6473","Type":"ContainerDied","Data":"4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463"} Nov 02 22:22:13.746018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.745916 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" event={"ID":"6784d5f2-659f-45cf-b603-99f19eae6473","Type":"ContainerDied","Data":"bf6c9f502aa104b17bc8aa679f3be1013ec77ec41ff53d50b9b677e26b7166aa"} Nov 02 22:22:13.746018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.745543 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7" Nov 02 22:22:13.746224 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:13.746201 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260\": container with ID starting with 660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260 not found: ID does not exist" containerID="660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260" Nov 02 22:22:13.746335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.746311 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260"} err="failed to get container status \"660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260\": rpc error: code = NotFound desc = could not find container \"660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260\": container with ID starting with 660740952f921de22dd4f5789da2303bf957d89d0a20390b44ceeabab06da260 not found: ID does not exist" Nov 02 22:22:13.746436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.746423 4285 scope.go:122] "RemoveContainer" containerID="af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f" Nov 02 22:22:13.758311 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.758290 4285 scope.go:122] "RemoveContainer" containerID="af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f" Nov 02 22:22:13.758566 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:13.758549 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f\": container with ID starting with af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f not found: ID does not exist" containerID="af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f" Nov 02 22:22:13.758619 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.758572 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f"} err="failed to get container status \"af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f\": rpc error: code = NotFound desc = could not find container \"af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f\": container with ID starting with af184ab3aeb68bf9d540a39537d22613905419210a8c11c27c36c967b0d6ba3f not found: ID does not exist" Nov 02 22:22:13.758619 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.758587 4285 scope.go:122] "RemoveContainer" containerID="335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb" Nov 02 22:22:13.767117 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.767099 4285 scope.go:122] "RemoveContainer" containerID="335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb" Nov 02 22:22:13.767390 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:13.767372 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb\": container with ID starting with 335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb not found: ID does not exist" containerID="335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb" Nov 02 22:22:13.767457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.767397 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb"} err="failed to get container status \"335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb\": rpc error: code = NotFound desc = could not find container \"335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb\": container with ID starting with 335de28281bac208e8ef4a1c42bb16e6043b001189b10ee00efd7222f585f0cb not found: ID does not exist" Nov 02 22:22:13.767457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.767418 4285 scope.go:122] "RemoveContainer" containerID="4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463" Nov 02 22:22:13.775921 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.775904 4285 scope.go:122] "RemoveContainer" containerID="4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463" Nov 02 22:22:13.776231 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:13.776210 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463\": container with ID starting with 4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463 not found: ID does not exist" containerID="4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463" Nov 02 22:22:13.776302 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:13.776239 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463"} err="failed to get container status \"4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463\": rpc error: code = NotFound desc = could not find container \"4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463\": container with ID starting with 4773089e24842123005df7d40cb87979cc7bf6e33ffbc4f840aa5da8ab10e463 not found: ID does not exist" Nov 02 22:22:14.006976 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.006905 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk"] Nov 02 22:22:14.010776 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.010751 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/premium-w4-10fdd5-kserve-84559c6446-g88lk"] Nov 02 22:22:14.118896 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.118862 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6"] Nov 02 22:22:14.123121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.123096 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-2-w4-10fdd5-kserve-778b8ff9cd-2b8t6"] Nov 02 22:22:14.235050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.234961 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk"] Nov 02 22:22:14.237137 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.237098 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/distinct-w4-10fdd5-kserve-754d654d89-mfzgk"] Nov 02 22:22:14.345262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.345231 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7"] Nov 02 22:22:14.349420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.349395 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-10fdd5/simulator-w4-10fdd5-kserve-cf85d484c-mb2p7"] Nov 02 22:22:14.446376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.446349 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:22:14.517336 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517308 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517369 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517397 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517433 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517459 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517516 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517540 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517591 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517699 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs\") pod \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\" (UID: \"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9\") " Nov 02 22:22:14.517765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517718 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:14.518289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.517902 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:14.518289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.518087 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:14.518289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.518099 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:14.518289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.518179 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.518289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.518204 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.519355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.519321 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:14.519944 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.519916 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:14.520203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.520181 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:14.520351 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.520329 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:22:14.520441 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.520425 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf" pod "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" (UID: "81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9"). InnerVolumeSpecName "kube-api-access-flsmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:14.618515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618442 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz"] Nov 02 22:22:14.618909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618884 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.618909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618910 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.619069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618920 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsmf\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-kube-api-access-flsmf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.619069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618933 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.619069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618944 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.619069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618952 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.619069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.618960 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:14.722768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.722736 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:14.754047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.754014 4285 generic.go:356] "Generic (PLEG): container finished" podID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerID="2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962" exitCode=0 Nov 02 22:22:14.754222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.754066 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" Nov 02 22:22:14.754222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.754099 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" event={"ID":"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9","Type":"ContainerDied","Data":"2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962"} Nov 02 22:22:14.754222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.754194 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw" event={"ID":"81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9","Type":"ContainerDied","Data":"de5aba00eab3eb5371fa1439a04e9f6d577cf6f15f587d6fb3a9eefed9ef219e"} Nov 02 22:22:14.754374 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.754221 4285 scope.go:122] "RemoveContainer" containerID="2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962" Nov 02 22:22:14.766958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.766939 4285 scope.go:122] "RemoveContainer" containerID="5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd" Nov 02 22:22:14.774796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.774780 4285 scope.go:122] "RemoveContainer" containerID="2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962" Nov 02 22:22:14.775062 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:14.775044 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962\": container with ID starting with 2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962 not found: ID does not exist" containerID="2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962" Nov 02 22:22:14.775121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.775070 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962"} err="failed to get container status \"2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962\": rpc error: code = NotFound desc = could not find container \"2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962\": container with ID starting with 2e8f5dcb78f10cf55b5d1d05a4b07536d609f73c725c250731c40472294b8962 not found: ID does not exist" Nov 02 22:22:14.775121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.775088 4285 scope.go:122] "RemoveContainer" containerID="5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd" Nov 02 22:22:14.775364 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:14.775346 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd\": container with ID starting with 5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd not found: ID does not exist" containerID="5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd" Nov 02 22:22:14.775412 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.775372 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd"} err="failed to get container status \"5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd\": rpc error: code = NotFound desc = could not find container \"5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd\": container with ID starting with 5e2e52b4c1bfa5f51dc7b1584b8114b7501845a7eccf1a9c128a729553b87fdd not found: ID does not exist" Nov 02 22:22:14.917653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.917595 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw"] Nov 02 22:22:14.919775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:14.919758 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-94a8a3-openshift-default-6d88867449-fs5fw"] Nov 02 22:22:15.172751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:15.172686 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" path="/var/lib/kubelet/pods/1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b/volumes" Nov 02 22:22:15.173081 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:15.173067 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" path="/var/lib/kubelet/pods/6784d5f2-659f-45cf-b603-99f19eae6473/volumes" Nov 02 22:22:15.173476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:15.173463 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" path="/var/lib/kubelet/pods/81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9/volumes" Nov 02 22:22:15.174230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:15.174215 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" path="/var/lib/kubelet/pods/e6a13596-0a3e-45e4-ab6e-25c06f8abb13/volumes" Nov 02 22:22:15.174565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:15.174554 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" path="/var/lib/kubelet/pods/fceaaaee-d6ec-41a0-92cd-63dad84030af/volumes" Nov 02 22:22:16.005167 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:16.005119 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:22:17.308713 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:17.308657 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:17.426750 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:17.426710 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:22:18.424594 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:18.424524 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:21.691667 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.691641 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:22:21.784477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.784404 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn\" (UniqueName: \"kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn\") pod \"ca086fab-21d1-4fc6-83de-2e36d41841d2\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " Nov 02 22:22:21.784477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.784441 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume\") pod \"ca086fab-21d1-4fc6-83de-2e36d41841d2\" (UID: \"ca086fab-21d1-4fc6-83de-2e36d41841d2\") " Nov 02 22:22:21.784853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.784834 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume" pod "ca086fab-21d1-4fc6-83de-2e36d41841d2" (UID: "ca086fab-21d1-4fc6-83de-2e36d41841d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:21.786503 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.786482 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn" pod "ca086fab-21d1-4fc6-83de-2e36d41841d2" (UID: "ca086fab-21d1-4fc6-83de-2e36d41841d2"). InnerVolumeSpecName "kube-api-access-z22rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:21.795380 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.795349 4285 generic.go:356] "Generic (PLEG): container finished" podID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerID="15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c" exitCode=0 Nov 02 22:22:21.795513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.795402 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" Nov 02 22:22:21.795513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.795447 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" event={"ID":"ca086fab-21d1-4fc6-83de-2e36d41841d2","Type":"ContainerDied","Data":"15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c"} Nov 02 22:22:21.795630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.795513 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd" event={"ID":"ca086fab-21d1-4fc6-83de-2e36d41841d2","Type":"ContainerDied","Data":"c335169ac1f30d090cddc740431e7feb1867a144c24b302b0138761b7be33fde"} Nov 02 22:22:21.795630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.795527 4285 scope.go:122] "RemoveContainer" containerID="15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c" Nov 02 22:22:21.807442 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.807371 4285 scope.go:122] "RemoveContainer" containerID="15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c" Nov 02 22:22:21.807669 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:21.807651 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c\": container with ID starting with 15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c not found: ID does not exist" containerID="15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c" Nov 02 22:22:21.807722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.807676 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c"} err="failed to get container status \"15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c\": rpc error: code = NotFound desc = could not find container \"15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c\": container with ID starting with 15d39d1120f2425ad67fc00b70a58caf00026eef9a4445c6b5717345c25dc98c not found: ID does not exist" Nov 02 22:22:21.885567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.885542 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z22rn\" (UniqueName: \"kubernetes.io/projected/ca086fab-21d1-4fc6-83de-2e36d41841d2-kube-api-access-z22rn\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:21.885567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.885568 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca086fab-21d1-4fc6-83de-2e36d41841d2-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:21.966078 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.966048 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd"] Nov 02 22:22:21.968046 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.968021 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:21.969484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:21.969463 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-94a8a3-b6b97dc9-g74bd"] Nov 02 22:22:22.120080 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:22.119992 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:23.173932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:23.173899 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" path="/var/lib/kubelet/pods/ca086fab-21d1-4fc6-83de-2e36d41841d2/volumes" Nov 02 22:22:28.099407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:28.099372 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.693796 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:28.693764 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.853695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:28.853657 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:29.638506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.638473 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" containerID="cri-o://41c48f35317758685117b7b0ef58b3039a7f891bdf7dae6a41ef0c17b89e1093" gracePeriod=45 Nov 02 22:22:29.833001 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.832967 4285 generic.go:356] "Generic (PLEG): container finished" podID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerID="41c48f35317758685117b7b0ef58b3039a7f891bdf7dae6a41ef0c17b89e1093" exitCode=0 Nov 02 22:22:29.833173 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.833077 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" event={"ID":"c9521ca7-741b-4807-bdde-0aa350f8f3a8","Type":"ContainerDied","Data":"41c48f35317758685117b7b0ef58b3039a7f891bdf7dae6a41ef0c17b89e1093"} Nov 02 22:22:29.886340 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.886316 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 22:22:29.962984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.962965 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm\") pod \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " Nov 02 22:22:29.963103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963020 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk\" (UniqueName: \"kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk\") pod \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " Nov 02 22:22:29.963103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963043 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir\") pod \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " Nov 02 22:22:29.963103 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963086 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache\") pod \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " Nov 02 22:22:29.963310 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963114 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home\") pod \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " Nov 02 22:22:29.963310 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963174 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs\") pod \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\" (UID: \"c9521ca7-741b-4807-bdde-0aa350f8f3a8\") " Nov 02 22:22:29.963609 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963379 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache" pod "c9521ca7-741b-4807-bdde-0aa350f8f3a8" (UID: "c9521ca7-741b-4807-bdde-0aa350f8f3a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:29.963728 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963397 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir" pod "c9521ca7-741b-4807-bdde-0aa350f8f3a8" (UID: "c9521ca7-741b-4807-bdde-0aa350f8f3a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:29.963728 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.963437 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home" pod "c9521ca7-741b-4807-bdde-0aa350f8f3a8" (UID: "c9521ca7-741b-4807-bdde-0aa350f8f3a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:29.965222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.965194 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm" pod "c9521ca7-741b-4807-bdde-0aa350f8f3a8" (UID: "c9521ca7-741b-4807-bdde-0aa350f8f3a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:29.965306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.965287 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs" pod "c9521ca7-741b-4807-bdde-0aa350f8f3a8" (UID: "c9521ca7-741b-4807-bdde-0aa350f8f3a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:29.965377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:29.965363 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk" pod "c9521ca7-741b-4807-bdde-0aa350f8f3a8" (UID: "c9521ca7-741b-4807-bdde-0aa350f8f3a8"). InnerVolumeSpecName "kube-api-access-wj4zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:30.064338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.064298 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4zk\" (UniqueName: \"kubernetes.io/projected/c9521ca7-741b-4807-bdde-0aa350f8f3a8-kube-api-access-wj4zk\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.064338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.064329 4285 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tmp-dir\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.064338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.064343 4285 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-model-cache\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.064557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.064354 4285 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-home\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.064557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.064368 4285 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9521ca7-741b-4807-bdde-0aa350f8f3a8-tls-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.064557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.064378 4285 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9521ca7-741b-4807-bdde-0aa350f8f3a8-dshm\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:30.838733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.838704 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" Nov 02 22:22:30.838733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.838720 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" event={"ID":"c9521ca7-741b-4807-bdde-0aa350f8f3a8","Type":"ContainerDied","Data":"075acbc68210a3e0db0bfd718530859afa2ef4683930611c7126399a641dc68d"} Nov 02 22:22:30.839217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.838766 4285 scope.go:122] "RemoveContainer" containerID="41c48f35317758685117b7b0ef58b3039a7f891bdf7dae6a41ef0c17b89e1093" Nov 02 22:22:30.841024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:30.840995 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 22:22:31.000948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.000915 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz"] Nov 02 22:22:31.007018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.006993 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9cc56b/premium-w2-9cc56b-kserve-56f4c6dfb-7sjcz"] Nov 02 22:22:31.067778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.067747 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz"] Nov 02 22:22:31.067970 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.067949 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" podUID="ad87c76d-6364-4168-963b-8c9ba2255a5a" containerName="maas-api" containerID="cri-o://137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a" gracePeriod=30 Nov 02 22:22:31.172903 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.172823 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" path="/var/lib/kubelet/pods/c9521ca7-741b-4807-bdde-0aa350f8f3a8/volumes" Nov 02 22:22:31.204991 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.204961 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:31.329477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.329453 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 22:22:31.487651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.487619 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs\") pod \"ad87c76d-6364-4168-963b-8c9ba2255a5a\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " Nov 02 22:22:31.487822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.487695 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2\" (UniqueName: \"kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2\") pod \"ad87c76d-6364-4168-963b-8c9ba2255a5a\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " Nov 02 22:22:31.487822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.487789 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\") pod \"ad87c76d-6364-4168-963b-8c9ba2255a5a\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " Nov 02 22:22:31.487890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.487843 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca\") pod \"ad87c76d-6364-4168-963b-8c9ba2255a5a\" (UID: \"ad87c76d-6364-4168-963b-8c9ba2255a5a\") " Nov 02 22:22:31.488217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.488190 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca" pod "ad87c76d-6364-4168-963b-8c9ba2255a5a" (UID: "ad87c76d-6364-4168-963b-8c9ba2255a5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:31.489712 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.489686 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2" pod "ad87c76d-6364-4168-963b-8c9ba2255a5a" (UID: "ad87c76d-6364-4168-963b-8c9ba2255a5a"). InnerVolumeSpecName "kube-api-access-q5bp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:31.490085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.490056 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls" pod "ad87c76d-6364-4168-963b-8c9ba2255a5a" (UID: "ad87c76d-6364-4168-963b-8c9ba2255a5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:31.490085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.490079 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs" pod "ad87c76d-6364-4168-963b-8c9ba2255a5a" (UID: "ad87c76d-6364-4168-963b-8c9ba2255a5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:31.588950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.588914 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bp2\" (UniqueName: \"kubernetes.io/projected/ad87c76d-6364-4168-963b-8c9ba2255a5a-kube-api-access-q5bp2\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.588950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.588944 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.589179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.588959 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad87c76d-6364-4168-963b-8c9ba2255a5a-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.589179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.588972 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad87c76d-6364-4168-963b-8c9ba2255a5a-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.845468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.845379 4285 generic.go:356] "Generic (PLEG): container finished" podID="ad87c76d-6364-4168-963b-8c9ba2255a5a" containerID="137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a" exitCode=0 Nov 02 22:22:31.845468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.845441 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" Nov 02 22:22:31.845982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.845479 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" event={"ID":"ad87c76d-6364-4168-963b-8c9ba2255a5a","Type":"ContainerDied","Data":"137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a"} Nov 02 22:22:31.845982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.845547 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz" event={"ID":"ad87c76d-6364-4168-963b-8c9ba2255a5a","Type":"ContainerDied","Data":"3dfdd94c796c74e6bd523809dd228b0fb42c18a45d11f7faf445ed798fb043cf"} Nov 02 22:22:31.845982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.845581 4285 scope.go:122] "RemoveContainer" containerID="137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a" Nov 02 22:22:31.862196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.857795 4285 scope.go:122] "RemoveContainer" containerID="137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a" Nov 02 22:22:31.862196 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:31.858171 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a\": container with ID starting with 137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a not found: ID does not exist" containerID="137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a" Nov 02 22:22:31.862196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:31.858208 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a"} err="failed to get container status \"137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a\": rpc error: code = NotFound desc = could not find container \"137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a\": container with ID starting with 137d467d084d574e4baa08faa3735c375f2a5c3a6497e81dda54b0ec4646ab1a not found: ID does not exist" Nov 02 22:22:32.007973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:32.007933 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz"] Nov 02 22:22:32.057272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:32.057238 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10fdd5-5b7b5c6cf7-kbfmz"] Nov 02 22:22:32.691990 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:32.691951 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:32.965913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:32.965812 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:33.172520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:33.172486 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad87c76d-6364-4168-963b-8c9ba2255a5a" path="/var/lib/kubelet/pods/ad87c76d-6364-4168-963b-8c9ba2255a5a/volumes" Nov 02 22:22:33.613253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:33.613198 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:35.461707 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:35.461660 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:35.627017 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:35.626981 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:36.136931 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:36.136901 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:39.783486 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:39.783452 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:39.953978 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:39.953945 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:40.103217 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.103144 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg"] Nov 02 22:22:40.103395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.103367 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" podUID="bbdb6075-2020-48d9-8fdd-82e4285a9eca" containerName="maas-api" containerID="cri-o://205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e" gracePeriod=30 Nov 02 22:22:40.125226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.125193 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:40.278160 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.278102 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:40.363176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.363090 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 22:22:40.469550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.469524 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\") pod \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " Nov 02 22:22:40.469684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.469560 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs\") pod \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " Nov 02 22:22:40.469735 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.469704 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn\" (UniqueName: \"kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn\") pod \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " Nov 02 22:22:40.469818 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.469805 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca\") pod \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\" (UID: \"bbdb6075-2020-48d9-8fdd-82e4285a9eca\") " Nov 02 22:22:40.470211 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.470177 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca" pod "bbdb6075-2020-48d9-8fdd-82e4285a9eca" (UID: "bbdb6075-2020-48d9-8fdd-82e4285a9eca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:40.471716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.471685 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs" pod "bbdb6075-2020-48d9-8fdd-82e4285a9eca" (UID: "bbdb6075-2020-48d9-8fdd-82e4285a9eca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:40.471823 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.471715 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls" pod "bbdb6075-2020-48d9-8fdd-82e4285a9eca" (UID: "bbdb6075-2020-48d9-8fdd-82e4285a9eca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:40.471823 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.471756 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn" pod "bbdb6075-2020-48d9-8fdd-82e4285a9eca" (UID: "bbdb6075-2020-48d9-8fdd-82e4285a9eca"). InnerVolumeSpecName "kube-api-access-zj4hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:40.571430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.571398 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj4hn\" (UniqueName: \"kubernetes.io/projected/bbdb6075-2020-48d9-8fdd-82e4285a9eca-kube-api-access-zj4hn\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:40.571430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.571426 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbdb6075-2020-48d9-8fdd-82e4285a9eca-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:40.571430 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.571437 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:40.571665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.571445 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbdb6075-2020-48d9-8fdd-82e4285a9eca-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:22:40.895914 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.895884 4285 generic.go:356] "Generic (PLEG): container finished" podID="bbdb6075-2020-48d9-8fdd-82e4285a9eca" containerID="205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e" exitCode=0 Nov 02 22:22:40.896349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.895927 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" Nov 02 22:22:40.896349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.895939 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" event={"ID":"bbdb6075-2020-48d9-8fdd-82e4285a9eca","Type":"ContainerDied","Data":"205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e"} Nov 02 22:22:40.896349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.895979 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg" event={"ID":"bbdb6075-2020-48d9-8fdd-82e4285a9eca","Type":"ContainerDied","Data":"a509daedb96e046776449d1133ff045ccbf45bcf49222fe24736b08b120b7a34"} Nov 02 22:22:40.896349 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.895999 4285 scope.go:122] "RemoveContainer" containerID="205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e" Nov 02 22:22:40.905861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.905841 4285 scope.go:122] "RemoveContainer" containerID="205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e" Nov 02 22:22:40.906152 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:40.906116 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e\": container with ID starting with 205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e not found: ID does not exist" containerID="205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e" Nov 02 22:22:40.906241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:40.906161 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e"} err="failed to get container status \"205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e\": rpc error: code = NotFound desc = could not find container \"205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e\": container with ID starting with 205d0106f83478d14df028cba8a2b547fd61503fd2d16fabc73c5e79f842354e not found: ID does not exist" Nov 02 22:22:41.059486 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:41.059450 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg"] Nov 02 22:22:41.108972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:41.108944 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9cc56b-54fbb6f6cf-6d9qg"] Nov 02 22:22:41.172299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:41.172229 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdb6075-2020-48d9-8fdd-82e4285a9eca" path="/var/lib/kubelet/pods/bbdb6075-2020-48d9-8fdd-82e4285a9eca/volumes" Nov 02 22:22:41.716610 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:41.716577 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:41.886209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:41.886176 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:42.037898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:42.037826 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:42.583928 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:42.583893 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:46.005156 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:22:46.005105 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:22:48.693724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:48.693690 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:48.877392 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:48.877350 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:49.038484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:49.038401 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:51.632197 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:51.632161 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:51.808032 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:51.807999 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:52.358024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:52.357994 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:57.866063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:57.866030 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:58.081586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:22:58.081554 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:15.563367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:15.563288 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:16.004195 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:23:16.004161 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:23:18.759329 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:18.759295 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:19.367279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367244 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls"] Nov 02 22:23:19.367617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367605 4285 cpu_manager.go:414] "containerMap: removing container" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" Nov 02 22:23:19.367661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367618 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" Nov 02 22:23:19.367661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367632 4285 cpu_manager.go:414] "containerMap: removing container" podUID="eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" containerName="maas-api" Nov 02 22:23:19.367661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367637 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" containerName="maas-api" Nov 02 22:23:19.367661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367645 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerName="payload-pre-processing" Nov 02 22:23:19.367661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367650 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerName="payload-pre-processing" Nov 02 22:23:19.367661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367660 4285 cpu_manager.go:414] "containerMap: removing container" podUID="3443b75b-8d6e-4388-828c-51520e369633" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367665 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3443b75b-8d6e-4388-828c-51520e369633" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367671 4285 cpu_manager.go:414] "containerMap: removing container" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367676 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367683 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ad87c76d-6364-4168-963b-8c9ba2255a5a" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367687 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad87c76d-6364-4168-963b-8c9ba2255a5a" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367693 4285 cpu_manager.go:414] "containerMap: removing container" podUID="8a24076f-b642-4d94-bb34-5211faf082ff" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367697 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a24076f-b642-4d94-bb34-5211faf082ff" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367705 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367709 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367716 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerName="payload-pre-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367720 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerName="payload-pre-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367727 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b1c42c48-894a-478b-858e-304f21ebcbff" containerName="payload-pre-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367732 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c42c48-894a-478b-858e-304f21ebcbff" containerName="payload-pre-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367741 4285 cpu_manager.go:414] "containerMap: removing container" podUID="c573ebd1-cb9d-4678-89ee-5afccf3b3954" containerName="payload-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367745 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c573ebd1-cb9d-4678-89ee-5afccf3b3954" containerName="payload-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367754 4285 cpu_manager.go:414] "containerMap: removing container" podUID="bbdb6075-2020-48d9-8fdd-82e4285a9eca" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367759 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdb6075-2020-48d9-8fdd-82e4285a9eca" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367766 4285 cpu_manager.go:414] "containerMap: removing container" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367771 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367778 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d2804ab9-431b-45f3-b8ef-23f7c27b2c51" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367782 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2804ab9-431b-45f3-b8ef-23f7c27b2c51" containerName="maas-api" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367791 4285 cpu_manager.go:414] "containerMap: removing container" podUID="33952884-fe06-4862-9077-7cfbc526fa76" containerName="payload-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367795 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33952884-fe06-4862-9077-7cfbc526fa76" containerName="payload-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367802 4285 cpu_manager.go:414] "containerMap: removing container" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367806 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367811 4285 cpu_manager.go:414] "containerMap: removing container" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367816 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367825 4285 cpu_manager.go:414] "containerMap: removing container" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367830 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367836 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367840 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367847 4285 cpu_manager.go:414] "containerMap: removing container" podUID="7058b902-fcd3-411b-af6b-d4d63b8a4cba" containerName="payload-pre-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367852 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7058b902-fcd3-411b-af6b-d4d63b8a4cba" containerName="payload-pre-processing" Nov 02 22:23:19.367845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367857 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" containerName="payload-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367862 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" containerName="payload-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367869 4285 cpu_manager.go:414] "containerMap: removing container" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367873 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367927 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="c573ebd1-cb9d-4678-89ee-5afccf3b3954" containerName="payload-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367935 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdb6075-2020-48d9-8fdd-82e4285a9eca" containerName="maas-api" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367941 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="847e7104-aadd-4903-835f-d71b8dce4beb" containerName="istio-proxy" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367947 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2804ab9-431b-45f3-b8ef-23f7c27b2c51" containerName="maas-api" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367953 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367958 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="33952884-fe06-4862-9077-7cfbc526fa76" containerName="payload-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367963 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9521ca7-741b-4807-bdde-0aa350f8f3a8" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367967 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e889220-94d6-4ee1-a77b-d0fed0c099be" containerName="istio-proxy" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367973 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="6784d5f2-659f-45cf-b603-99f19eae6473" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367978 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a13596-0a3e-45e4-ab6e-25c06f8abb13" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367984 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="7058b902-fcd3-411b-af6b-d4d63b8a4cba" containerName="payload-pre-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367988 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a92a95-a4bf-4805-a8ce-e2b0c0206c87" containerName="payload-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367994 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde78196-f98e-444a-9419-25e38531db04" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.367999 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="fceaaaee-d6ec-41a0-92cd-63dad84030af" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368004 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb47c9e0-ddf8-4953-ade9-1d2bfeeec869" containerName="maas-api" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368008 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f54f65-697b-43af-abbe-f1e530906d7d" containerName="payload-pre-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368014 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="3443b75b-8d6e-4388-828c-51520e369633" containerName="maas-api" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368019 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad87c76d-6364-4168-963b-8c9ba2255a5a" containerName="maas-api" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368024 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a24076f-b642-4d94-bb34-5211faf082ff" containerName="maas-api" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368029 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb17e5a-fbf5-4fc0-b8e7-e51a0ae3be7b" containerName="main" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368034 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca086fab-21d1-4fc6-83de-2e36d41841d2" containerName="payload-pre-processing" Nov 02 22:23:19.368843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.368038 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c42c48-894a-478b-858e-304f21ebcbff" containerName="payload-pre-processing" Nov 02 22:23:19.371155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.371139 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.427918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.427885 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls"] Nov 02 22:23:19.475037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.475011 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:23:19.475037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.475011 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:23:19.513620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.513587 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkfd\" (UniqueName: \"kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.513752 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.513658 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.513797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.513753 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.614845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.614814 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.614999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.614882 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkfd\" (UniqueName: \"kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.614999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.614936 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.615586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.615561 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.617156 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.617105 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.671231 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.671177 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkfd\" (UniqueName: \"kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd\") pod \"maas-api-revoke-sub-85237e45-txqls\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.682998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.682978 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:19.787859 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.787834 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx"] Nov 02 22:23:19.788242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.788229 4285 cpu_manager.go:414] "containerMap: removing container" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" Nov 02 22:23:19.788291 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.788242 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" Nov 02 22:23:19.788328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.788311 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e0ee1c-daab-40ec-9d4e-8ffc6e4ce8c9" containerName="istio-proxy" Nov 02 22:23:19.792376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.792361 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.916966 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917061 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxvf\" (UniqueName: \"kubernetes.io/projected/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-kube-api-access-pwxvf\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917180 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917316 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917220 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917316 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917266 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917316 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917289 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917316 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917312 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917485 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917378 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:19.917485 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:19.917427 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.008218 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.008115 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx"] Nov 02 22:23:20.018683 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018653 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.018844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018687 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.018844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018702 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.018844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018740 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.018844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018780 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.018844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018829 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.018960 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxvf\" (UniqueName: \"kubernetes.io/projected/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-kube-api-access-pwxvf\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019039 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019068 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019137 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019250 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019278 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019422 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.019789 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.019769 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.021149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.021117 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.021715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.021697 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.060002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.059981 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls"] Nov 02 22:23:20.062023 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:23:20.061999 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2994c34d_c7fb_47f7_bc31_a583dc8473c6.slice/crio-e1f713f7a09925c986e322b8c64b5743a34c58b02a7b4fff68924a418ba74bc6 WatchSource:0}: Error finding container e1f713f7a09925c986e322b8c64b5743a34c58b02a7b4fff68924a418ba74bc6: Status 404 returned error can't find the container with id e1f713f7a09925c986e322b8c64b5743a34c58b02a7b4fff68924a418ba74bc6 Nov 02 22:23:20.075477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.075459 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.075551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.075525 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxvf\" (UniqueName: \"kubernetes.io/projected/dc0ddbc9-69dd-411e-bf0f-423369dc5b2d-kube-api-access-pwxvf\") pod \"maas-default-gateway-openshift-default-65c448b547-sf2hx\" (UID: \"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.076998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.076973 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" event={"ID":"2994c34d-c7fb-47f7-bc31-a583dc8473c6","Type":"ContainerStarted","Data":"e1f713f7a09925c986e322b8c64b5743a34c58b02a7b4fff68924a418ba74bc6"} Nov 02 22:23:20.104916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.104897 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:20.524068 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.524045 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx"] Nov 02 22:23:20.526537 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:23:20.526509 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0ddbc9_69dd_411e_bf0f_423369dc5b2d.slice/crio-36a8b0e5223d70b82e2f934c81113c269bfd048f500017755be3bedf2ec0cbab WatchSource:0}: Error finding container 36a8b0e5223d70b82e2f934c81113c269bfd048f500017755be3bedf2ec0cbab: Status 404 returned error can't find the container with id 36a8b0e5223d70b82e2f934c81113c269bfd048f500017755be3bedf2ec0cbab Nov 02 22:23:20.528604 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.528573 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:20.528686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.528633 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:20.528686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:20.528660 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:21.082599 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.082565 4285 generic.go:356] "Generic (PLEG): container finished" podID="2994c34d-c7fb-47f7-bc31-a583dc8473c6" containerID="86840346638c065fa3f796f0692e6c6e3c7760b5547c2aa91b1c3f6f252a7599" exitCode=0 Nov 02 22:23:21.083002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.082613 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" event={"ID":"2994c34d-c7fb-47f7-bc31-a583dc8473c6","Type":"ContainerDied","Data":"86840346638c065fa3f796f0692e6c6e3c7760b5547c2aa91b1c3f6f252a7599"} Nov 02 22:23:21.084208 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.084182 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" event={"ID":"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d","Type":"ContainerStarted","Data":"d29dc92e5c6554a7e19abd27ccb67674c74ee857c539317a02df0ce6775ac0d4"} Nov 02 22:23:21.084309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.084226 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" event={"ID":"dc0ddbc9-69dd-411e-bf0f-423369dc5b2d","Type":"ContainerStarted","Data":"36a8b0e5223d70b82e2f934c81113c269bfd048f500017755be3bedf2ec0cbab"} Nov 02 22:23:21.105672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.105649 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:21.107210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.107191 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:21.107290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.107241 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:21.370143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:21.370082 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podStartSLOduration=2.370067845 podStartE2EDuration="2.370067845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:21.369439925 +0000 UTC m=+2940.330990244" watchObservedRunningTime="2026-11-02 22:23:21.370067845 +0000 UTC m=+2940.331618164" Nov 02 22:23:22.105950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.105919 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:22.106408 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.105973 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:22.259765 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.259739 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:22.351258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.351190 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca\") pod \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " Nov 02 22:23:22.351258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.351243 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca\") pod \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " Nov 02 22:23:22.351424 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.351315 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd\" (UniqueName: \"kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd\") pod \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\" (UID: \"2994c34d-c7fb-47f7-bc31-a583dc8473c6\") " Nov 02 22:23:22.351495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.351474 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca" pod "2994c34d-c7fb-47f7-bc31-a583dc8473c6" (UID: "2994c34d-c7fb-47f7-bc31-a583dc8473c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:22.351603 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.351584 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:23:22.353342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.353311 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca" pod "2994c34d-c7fb-47f7-bc31-a583dc8473c6" (UID: "2994c34d-c7fb-47f7-bc31-a583dc8473c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:23:22.353427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.353326 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd" pod "2994c34d-c7fb-47f7-bc31-a583dc8473c6" (UID: "2994c34d-c7fb-47f7-bc31-a583dc8473c6"). InnerVolumeSpecName "kube-api-access-brkfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:22.452689 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.452662 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkfd\" (UniqueName: \"kubernetes.io/projected/2994c34d-c7fb-47f7-bc31-a583dc8473c6-kube-api-access-brkfd\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:23:22.452689 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:22.452687 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2994c34d-c7fb-47f7-bc31-a583dc8473c6-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:23:23.097538 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:23.097505 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" Nov 02 22:23:23.097710 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:23.097534 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls" event={"ID":"2994c34d-c7fb-47f7-bc31-a583dc8473c6","Type":"ContainerDied","Data":"e1f713f7a09925c986e322b8c64b5743a34c58b02a7b4fff68924a418ba74bc6"} Nov 02 22:23:23.097710 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:23.097580 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1f713f7a09925c986e322b8c64b5743a34c58b02a7b4fff68924a418ba74bc6" Nov 02 22:23:23.105336 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:23.105316 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:23.105435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:23.105355 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:24.105519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:24.105481 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:24.105963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:24.105531 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:25.105758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:25.105720 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:25.106159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:25.105780 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:26.105764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:26.105726 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:26.106175 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:26.105785 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:27.105506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:27.105461 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:23:27.105689 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:27.105547 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 22:23:29.106309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:29.106264 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:23:29.106808 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:29.106319 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:23:30.148288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:30.148242 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-sf2hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:23:30.148804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:30.148309 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" podUID="dc0ddbc9-69dd-411e-bf0f-423369dc5b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:23:30.440958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:30.440882 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:30.441172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:30.441152 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:30.441838 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:30.441817 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-sf2hx" Nov 02 22:23:31.848479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:31.848444 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:33.617615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:33.617581 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:33.766190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:33.766162 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:34.187333 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:23:34.187297 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:46.003814 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:23:46.003772 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:24:16.005139 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:24:16.005085 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:24:19.091347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:19.091315 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:20.880442 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:20.880414 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:27.862309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.862277 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658"] Nov 02 22:24:27.862757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.862682 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2994c34d-c7fb-47f7-bc31-a583dc8473c6" containerName="revoke-keys" Nov 02 22:24:27.862757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.862693 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2994c34d-c7fb-47f7-bc31-a583dc8473c6" containerName="revoke-keys" Nov 02 22:24:27.862757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.862758 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2994c34d-c7fb-47f7-bc31-a583dc8473c6" containerName="revoke-keys" Nov 02 22:24:27.865520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.865503 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:27.920481 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.920454 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:27.941144 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.941103 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd849\" (UniqueName: \"kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:27.941265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.941211 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:27.941265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.941249 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:27.970027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.970002 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:24:27.971924 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:27.971899 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658"] Nov 02 22:24:28.042182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.042150 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.042285 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.042243 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd849\" (UniqueName: \"kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.042285 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.042282 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.042817 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.042794 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.044407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.044384 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.098359 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.098335 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd849\" (UniqueName: \"kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849\") pod \"maas-api-revoke-sub-c1e271a9-5g658\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.177448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.177399 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:28.543513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:28.543489 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658"] Nov 02 22:24:28.545156 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:24:28.545113 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cb211d_5a41_4487_a5f9_3d8eb601c13e.slice/crio-3514e4c5e6a671fd7596dc37dedd5a53d6336091a1ff2db50a6af68a83d9de03 WatchSource:0}: Error finding container 3514e4c5e6a671fd7596dc37dedd5a53d6336091a1ff2db50a6af68a83d9de03: Status 404 returned error can't find the container with id 3514e4c5e6a671fd7596dc37dedd5a53d6336091a1ff2db50a6af68a83d9de03 Nov 02 22:24:29.426330 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:29.426304 4285 generic.go:356] "Generic (PLEG): container finished" podID="54cb211d-5a41-4487-a5f9-3d8eb601c13e" containerID="eb77b94904715f3dab00aa3436022f48b41be2e61a6a3a6e64512ac85a6b1453" exitCode=0 Nov 02 22:24:29.426676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:29.426394 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" event={"ID":"54cb211d-5a41-4487-a5f9-3d8eb601c13e","Type":"ContainerDied","Data":"eb77b94904715f3dab00aa3436022f48b41be2e61a6a3a6e64512ac85a6b1453"} Nov 02 22:24:29.426676 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:29.426448 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" event={"ID":"54cb211d-5a41-4487-a5f9-3d8eb601c13e","Type":"ContainerStarted","Data":"3514e4c5e6a671fd7596dc37dedd5a53d6336091a1ff2db50a6af68a83d9de03"} Nov 02 22:24:30.556154 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.556116 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:30.561398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.561376 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca\") pod \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " Nov 02 22:24:30.561540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.561470 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca\") pod \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " Nov 02 22:24:30.561540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.561508 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849\" (UniqueName: \"kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849\") pod \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\" (UID: \"54cb211d-5a41-4487-a5f9-3d8eb601c13e\") " Nov 02 22:24:30.561806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.561785 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca" pod "54cb211d-5a41-4487-a5f9-3d8eb601c13e" (UID: "54cb211d-5a41-4487-a5f9-3d8eb601c13e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:30.563433 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.563411 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca" pod "54cb211d-5a41-4487-a5f9-3d8eb601c13e" (UID: "54cb211d-5a41-4487-a5f9-3d8eb601c13e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:30.563534 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.563414 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849" pod "54cb211d-5a41-4487-a5f9-3d8eb601c13e" (UID: "54cb211d-5a41-4487-a5f9-3d8eb601c13e"). InnerVolumeSpecName "kube-api-access-bd849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:30.662800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.662775 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:24:30.662800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.662798 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cb211d-5a41-4487-a5f9-3d8eb601c13e-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:24:30.662962 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:30.662810 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd849\" (UniqueName: \"kubernetes.io/projected/54cb211d-5a41-4487-a5f9-3d8eb601c13e-kube-api-access-bd849\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:24:31.435583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:31.435509 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" event={"ID":"54cb211d-5a41-4487-a5f9-3d8eb601c13e","Type":"ContainerDied","Data":"3514e4c5e6a671fd7596dc37dedd5a53d6336091a1ff2db50a6af68a83d9de03"} Nov 02 22:24:31.435583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:31.435542 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658" Nov 02 22:24:31.435583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:31.435554 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3514e4c5e6a671fd7596dc37dedd5a53d6336091a1ff2db50a6af68a83d9de03" Nov 02 22:24:46.005874 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:24:46.005781 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:24:59.146616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.146582 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7"] Nov 02 22:24:59.147090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.146959 4285 cpu_manager.go:414] "containerMap: removing container" podUID="54cb211d-5a41-4487-a5f9-3d8eb601c13e" containerName="revoke-keys" Nov 02 22:24:59.147090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.146970 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cb211d-5a41-4487-a5f9-3d8eb601c13e" containerName="revoke-keys" Nov 02 22:24:59.147090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.147039 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cb211d-5a41-4487-a5f9-3d8eb601c13e" containerName="revoke-keys" Nov 02 22:24:59.150252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.150236 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.254708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.254676 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:59.254708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.254685 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:24:59.256751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.256732 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7"] Nov 02 22:24:59.257115 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.257092 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xrz\" (UniqueName: \"kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.257286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.257270 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.257341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.257305 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.358304 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.358280 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xrz\" (UniqueName: \"kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.358462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.358365 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.358462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.358393 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.358997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.358978 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.360582 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.360560 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.414097 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.414043 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xrz\" (UniqueName: \"kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz\") pod \"maas-api-revoke-sub-2e1f5b5f-zk4c7\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.463234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.463214 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:24:59.829449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:24:59.829427 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7"] Nov 02 22:24:59.831567 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:24:59.831541 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a6e76b_06e1_4590_8d4e_33f66724f2ea.slice/crio-fd1e16e5eb71b0de897ef8735cc97a4ade75c1f995d381573eba859a2ee6cc0c WatchSource:0}: Error finding container fd1e16e5eb71b0de897ef8735cc97a4ade75c1f995d381573eba859a2ee6cc0c: Status 404 returned error can't find the container with id fd1e16e5eb71b0de897ef8735cc97a4ade75c1f995d381573eba859a2ee6cc0c Nov 02 22:25:00.576649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:00.576608 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" event={"ID":"38a6e76b-06e1-4590-8d4e-33f66724f2ea","Type":"ContainerStarted","Data":"a537965a00ec293eb9d9b578f856da90952c2d426b06e8c88c031e74ef6f3f4c"} Nov 02 22:25:00.577029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:00.576702 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" event={"ID":"38a6e76b-06e1-4590-8d4e-33f66724f2ea","Type":"ContainerStarted","Data":"fd1e16e5eb71b0de897ef8735cc97a4ade75c1f995d381573eba859a2ee6cc0c"} Nov 02 22:25:00.690339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:00.690296 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" podStartSLOduration=1.690282295 podStartE2EDuration="1.690282295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:00.688973898 +0000 UTC m=+3039.650524219" watchObservedRunningTime="2026-11-02 22:25:00.690282295 +0000 UTC m=+3039.651832611" Nov 02 22:25:01.582172 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:01.582112 4285 generic.go:356] "Generic (PLEG): container finished" podID="38a6e76b-06e1-4590-8d4e-33f66724f2ea" containerID="a537965a00ec293eb9d9b578f856da90952c2d426b06e8c88c031e74ef6f3f4c" exitCode=0 Nov 02 22:25:01.582546 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:01.582197 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" event={"ID":"38a6e76b-06e1-4590-8d4e-33f66724f2ea","Type":"ContainerDied","Data":"a537965a00ec293eb9d9b578f856da90952c2d426b06e8c88c031e74ef6f3f4c"} Nov 02 22:25:02.715219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.715195 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:25:02.893962 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.893895 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz\" (UniqueName: \"kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz\") pod \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " Nov 02 22:25:02.894104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.893997 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca\") pod \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " Nov 02 22:25:02.894104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.894073 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca\") pod \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\" (UID: \"38a6e76b-06e1-4590-8d4e-33f66724f2ea\") " Nov 02 22:25:02.894429 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.894405 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca" pod "38a6e76b-06e1-4590-8d4e-33f66724f2ea" (UID: "38a6e76b-06e1-4590-8d4e-33f66724f2ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:02.895853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.895827 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz" pod "38a6e76b-06e1-4590-8d4e-33f66724f2ea" (UID: "38a6e76b-06e1-4590-8d4e-33f66724f2ea"). InnerVolumeSpecName "kube-api-access-m7xrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:02.895956 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.895889 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca" pod "38a6e76b-06e1-4590-8d4e-33f66724f2ea" (UID: "38a6e76b-06e1-4590-8d4e-33f66724f2ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:02.994875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.994850 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:02.994875 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.994872 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a6e76b-06e1-4590-8d4e-33f66724f2ea-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:02.995010 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:02.994884 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xrz\" (UniqueName: \"kubernetes.io/projected/38a6e76b-06e1-4590-8d4e-33f66724f2ea-kube-api-access-m7xrz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:03.592519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:03.592489 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" Nov 02 22:25:03.592686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:03.592519 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7" event={"ID":"38a6e76b-06e1-4590-8d4e-33f66724f2ea","Type":"ContainerDied","Data":"fd1e16e5eb71b0de897ef8735cc97a4ade75c1f995d381573eba859a2ee6cc0c"} Nov 02 22:25:03.592686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:03.592567 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd1e16e5eb71b0de897ef8735cc97a4ade75c1f995d381573eba859a2ee6cc0c" Nov 02 22:25:11.540164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.540119 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh"] Nov 02 22:25:11.540568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.540542 4285 cpu_manager.go:414] "containerMap: removing container" podUID="38a6e76b-06e1-4590-8d4e-33f66724f2ea" containerName="revoke-keys" Nov 02 22:25:11.540568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.540552 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a6e76b-06e1-4590-8d4e-33f66724f2ea" containerName="revoke-keys" Nov 02 22:25:11.540653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.540625 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a6e76b-06e1-4590-8d4e-33f66724f2ea" containerName="revoke-keys" Nov 02 22:25:11.543709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.543691 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.568058 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.568029 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.568209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.568077 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqpq\" (UniqueName: \"kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.568327 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.568302 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.599373 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.599274 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:11.648778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.648747 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:25:11.650767 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.650745 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh"] Nov 02 22:25:11.669036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.669003 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.669216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.669097 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.669216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.669162 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqpq\" (UniqueName: \"kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.669743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.669722 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.671245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.671224 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.725408 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.725387 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqpq\" (UniqueName: \"kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq\") pod \"maas-api-revoke-sub-ec843018-h9tjh\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:11.856650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:11.856591 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:12.227693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:12.227666 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh"] Nov 02 22:25:12.229556 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:25:12.229527 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf34952f_37c4_4aff_a711_14614a92ef5c.slice/crio-c449382a1bac5139ba642f299ee9a5be175699992085d014ce01218dcd191aef WatchSource:0}: Error finding container c449382a1bac5139ba642f299ee9a5be175699992085d014ce01218dcd191aef: Status 404 returned error can't find the container with id c449382a1bac5139ba642f299ee9a5be175699992085d014ce01218dcd191aef Nov 02 22:25:12.635198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:12.635164 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" event={"ID":"df34952f-37c4-4aff-a711-14614a92ef5c","Type":"ContainerStarted","Data":"d1a4c14a15bf2787a9eb7debf24ccac2c7bd6ad9419ec79f24db6f322c38fe51"} Nov 02 22:25:12.635536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:12.635228 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" event={"ID":"df34952f-37c4-4aff-a711-14614a92ef5c","Type":"ContainerStarted","Data":"c449382a1bac5139ba642f299ee9a5be175699992085d014ce01218dcd191aef"} Nov 02 22:25:12.750658 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:12.750607 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" podStartSLOduration=1.750594276 podStartE2EDuration="1.750594276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:12.749648809 +0000 UTC m=+3051.711199128" watchObservedRunningTime="2026-11-02 22:25:12.750594276 +0000 UTC m=+3051.712144594" Nov 02 22:25:13.641111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:13.641078 4285 generic.go:356] "Generic (PLEG): container finished" podID="df34952f-37c4-4aff-a711-14614a92ef5c" containerID="d1a4c14a15bf2787a9eb7debf24ccac2c7bd6ad9419ec79f24db6f322c38fe51" exitCode=0 Nov 02 22:25:13.641517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:13.641154 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" event={"ID":"df34952f-37c4-4aff-a711-14614a92ef5c","Type":"ContainerDied","Data":"d1a4c14a15bf2787a9eb7debf24ccac2c7bd6ad9419ec79f24db6f322c38fe51"} Nov 02 22:25:14.773607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.773583 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:14.796974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.796947 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca\") pod \"df34952f-37c4-4aff-a711-14614a92ef5c\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " Nov 02 22:25:14.797105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.797000 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca\") pod \"df34952f-37c4-4aff-a711-14614a92ef5c\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " Nov 02 22:25:14.797105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.797074 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq\" (UniqueName: \"kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq\") pod \"df34952f-37c4-4aff-a711-14614a92ef5c\" (UID: \"df34952f-37c4-4aff-a711-14614a92ef5c\") " Nov 02 22:25:14.797347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.797322 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca" pod "df34952f-37c4-4aff-a711-14614a92ef5c" (UID: "df34952f-37c4-4aff-a711-14614a92ef5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:14.798929 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.798904 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca" pod "df34952f-37c4-4aff-a711-14614a92ef5c" (UID: "df34952f-37c4-4aff-a711-14614a92ef5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:14.799025 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.799001 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq" pod "df34952f-37c4-4aff-a711-14614a92ef5c" (UID: "df34952f-37c4-4aff-a711-14614a92ef5c"). InnerVolumeSpecName "kube-api-access-7qqpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:14.898799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.898736 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:14.898799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.898768 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df34952f-37c4-4aff-a711-14614a92ef5c-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:14.898799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:14.898777 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqpq\" (UniqueName: \"kubernetes.io/projected/df34952f-37c4-4aff-a711-14614a92ef5c-kube-api-access-7qqpq\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:15.651861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:15.651829 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" event={"ID":"df34952f-37c4-4aff-a711-14614a92ef5c","Type":"ContainerDied","Data":"c449382a1bac5139ba642f299ee9a5be175699992085d014ce01218dcd191aef"} Nov 02 22:25:15.652029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:15.651877 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c449382a1bac5139ba642f299ee9a5be175699992085d014ce01218dcd191aef" Nov 02 22:25:15.652029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:15.651845 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh" Nov 02 22:25:16.004690 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:25:16.004657 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:25:19.474569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:19.474529 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:46.004151 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:25:46.004100 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:25:49.346642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.346609 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l"] Nov 02 22:25:49.347014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.346970 4285 cpu_manager.go:414] "containerMap: removing container" podUID="df34952f-37c4-4aff-a711-14614a92ef5c" containerName="revoke-keys" Nov 02 22:25:49.347014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.346980 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df34952f-37c4-4aff-a711-14614a92ef5c" containerName="revoke-keys" Nov 02 22:25:49.347090 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.347063 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="df34952f-37c4-4aff-a711-14614a92ef5c" containerName="revoke-keys" Nov 02 22:25:49.350071 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.350056 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.407640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.407618 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l"] Nov 02 22:25:49.429663 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.429643 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.429800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.429686 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwmw\" (UniqueName: \"kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.429800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.429712 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.451238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.451218 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:49.451331 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.451259 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:25:49.530345 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.530321 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.530558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.530369 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwmw\" (UniqueName: \"kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.530558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.530396 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.531021 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.531003 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.532544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.532521 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.586258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.586237 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwmw\" (UniqueName: \"kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw\") pod \"maas-api-revoke-sub-046e1343-zzm5l\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:49.662155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:49.662081 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:50.027854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:50.027830 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l"] Nov 02 22:25:50.835861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:50.835833 4285 generic.go:356] "Generic (PLEG): container finished" podID="a50d6d67-233d-44a7-be9c-7d42bf29ba91" containerID="0e22638d521026c24fdbf1375cf7acd8215b0e758e0e2183e312e273d3e6054f" exitCode=0 Nov 02 22:25:50.836187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:50.835910 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" event={"ID":"a50d6d67-233d-44a7-be9c-7d42bf29ba91","Type":"ContainerDied","Data":"0e22638d521026c24fdbf1375cf7acd8215b0e758e0e2183e312e273d3e6054f"} Nov 02 22:25:50.836187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:50.835961 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" event={"ID":"a50d6d67-233d-44a7-be9c-7d42bf29ba91","Type":"ContainerStarted","Data":"b28074ee9ff93cf9ca9b321c23492ddb09420fbfad0b84af30ffbbafca98286d"} Nov 02 22:25:51.977478 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:51.977454 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:52.054366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.054345 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw\" (UniqueName: \"kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw\") pod \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " Nov 02 22:25:52.054468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.054399 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca\") pod \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " Nov 02 22:25:52.054468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.054446 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca\") pod \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\" (UID: \"a50d6d67-233d-44a7-be9c-7d42bf29ba91\") " Nov 02 22:25:52.054825 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.054799 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca" pod "a50d6d67-233d-44a7-be9c-7d42bf29ba91" (UID: "a50d6d67-233d-44a7-be9c-7d42bf29ba91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:52.056253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.056233 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca" pod "a50d6d67-233d-44a7-be9c-7d42bf29ba91" (UID: "a50d6d67-233d-44a7-be9c-7d42bf29ba91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:52.056327 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.056251 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw" pod "a50d6d67-233d-44a7-be9c-7d42bf29ba91" (UID: "a50d6d67-233d-44a7-be9c-7d42bf29ba91"). InnerVolumeSpecName "kube-api-access-zjwmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:52.155908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.155853 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjwmw\" (UniqueName: \"kubernetes.io/projected/a50d6d67-233d-44a7-be9c-7d42bf29ba91-kube-api-access-zjwmw\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:52.155908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.155877 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:52.155908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.155887 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50d6d67-233d-44a7-be9c-7d42bf29ba91-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:25:52.846185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.846157 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" Nov 02 22:25:52.846352 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.846157 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l" event={"ID":"a50d6d67-233d-44a7-be9c-7d42bf29ba91","Type":"ContainerDied","Data":"b28074ee9ff93cf9ca9b321c23492ddb09420fbfad0b84af30ffbbafca98286d"} Nov 02 22:25:52.846352 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:25:52.846284 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28074ee9ff93cf9ca9b321c23492ddb09420fbfad0b84af30ffbbafca98286d" Nov 02 22:26:00.071549 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.071507 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5"] Nov 02 22:26:00.072022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.071881 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a50d6d67-233d-44a7-be9c-7d42bf29ba91" containerName="revoke-keys" Nov 02 22:26:00.072022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.071892 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50d6d67-233d-44a7-be9c-7d42bf29ba91" containerName="revoke-keys" Nov 02 22:26:00.072022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.071968 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50d6d67-233d-44a7-be9c-7d42bf29ba91" containerName="revoke-keys" Nov 02 22:26:00.075555 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.075529 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.119920 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.119890 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.120072 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.119943 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkw2z\" (UniqueName: \"kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.120072 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.119960 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.179292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.179259 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:00.179292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.179270 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:26:00.181500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.181473 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5"] Nov 02 22:26:00.221166 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.221138 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkw2z\" (UniqueName: \"kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.221311 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.221172 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.221460 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.221439 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.221795 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.221774 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.223577 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.223555 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.277232 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.277206 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkw2z\" (UniqueName: \"kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z\") pod \"maas-api-revoke-sub-d3fba056-rjxz5\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.388716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.388656 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:00.754844 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.754819 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5"] Nov 02 22:26:00.756625 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:26:00.756599 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8294d0b_2444_4a8b_993f_3e104e443c81.slice/crio-b4cbf156d4df5194bb85aa9ed23d001876930b32879b2fde979b7e0236694e35 WatchSource:0}: Error finding container b4cbf156d4df5194bb85aa9ed23d001876930b32879b2fde979b7e0236694e35: Status 404 returned error can't find the container with id b4cbf156d4df5194bb85aa9ed23d001876930b32879b2fde979b7e0236694e35 Nov 02 22:26:00.885347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:00.885322 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" event={"ID":"e8294d0b-2444-4a8b-993f-3e104e443c81","Type":"ContainerStarted","Data":"b4cbf156d4df5194bb85aa9ed23d001876930b32879b2fde979b7e0236694e35"} Nov 02 22:26:01.552419 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:26:01.552387 4285 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8294d0b_2444_4a8b_993f_3e104e443c81.slice/crio-65498b90e12facd8f1824e1db1e4402939168a30348e1af3fcfa6e2510c4ebc1.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:01.891701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:01.891615 4285 generic.go:356] "Generic (PLEG): container finished" podID="e8294d0b-2444-4a8b-993f-3e104e443c81" containerID="65498b90e12facd8f1824e1db1e4402939168a30348e1af3fcfa6e2510c4ebc1" exitCode=0 Nov 02 22:26:01.891846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:01.891701 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" event={"ID":"e8294d0b-2444-4a8b-993f-3e104e443c81","Type":"ContainerDied","Data":"65498b90e12facd8f1824e1db1e4402939168a30348e1af3fcfa6e2510c4ebc1"} Nov 02 22:26:03.071205 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.071180 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:03.150423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.150332 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z\" (UniqueName: \"kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z\") pod \"e8294d0b-2444-4a8b-993f-3e104e443c81\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " Nov 02 22:26:03.150570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.150432 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca\") pod \"e8294d0b-2444-4a8b-993f-3e104e443c81\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " Nov 02 22:26:03.150570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.150552 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca\") pod \"e8294d0b-2444-4a8b-993f-3e104e443c81\" (UID: \"e8294d0b-2444-4a8b-993f-3e104e443c81\") " Nov 02 22:26:03.151001 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.150970 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca" pod "e8294d0b-2444-4a8b-993f-3e104e443c81" (UID: "e8294d0b-2444-4a8b-993f-3e104e443c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:03.152594 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.152561 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z" pod "e8294d0b-2444-4a8b-993f-3e104e443c81" (UID: "e8294d0b-2444-4a8b-993f-3e104e443c81"). InnerVolumeSpecName "kube-api-access-xkw2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:03.153479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.153452 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca" pod "e8294d0b-2444-4a8b-993f-3e104e443c81" (UID: "e8294d0b-2444-4a8b-993f-3e104e443c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:03.252407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.252367 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkw2z\" (UniqueName: \"kubernetes.io/projected/e8294d0b-2444-4a8b-993f-3e104e443c81-kube-api-access-xkw2z\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.252407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.252409 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.252647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.252426 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8294d0b-2444-4a8b-993f-3e104e443c81-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.904295 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.904220 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" Nov 02 22:26:03.904532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.904219 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5" event={"ID":"e8294d0b-2444-4a8b-993f-3e104e443c81","Type":"ContainerDied","Data":"b4cbf156d4df5194bb85aa9ed23d001876930b32879b2fde979b7e0236694e35"} Nov 02 22:26:03.904532 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:03.904357 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4cbf156d4df5194bb85aa9ed23d001876930b32879b2fde979b7e0236694e35" Nov 02 22:26:16.004612 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:26:16.004578 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:26:19.772288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:19.772237 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:34.607990 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.607957 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55"] Nov 02 22:26:34.608384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.608342 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e8294d0b-2444-4a8b-993f-3e104e443c81" containerName="revoke-keys" Nov 02 22:26:34.608384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.608355 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8294d0b-2444-4a8b-993f-3e104e443c81" containerName="revoke-keys" Nov 02 22:26:34.608457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.608428 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8294d0b-2444-4a8b-993f-3e104e443c81" containerName="revoke-keys" Nov 02 22:26:34.611255 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.611240 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.715270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.715232 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:34.715420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.715354 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:26:34.717428 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.717408 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55"] Nov 02 22:26:34.791833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.791809 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzfv\" (UniqueName: \"kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.791974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.791841 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.791974 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.791953 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.893836 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.893770 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzfv\" (UniqueName: \"kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.894162 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.894118 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.894373 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.894355 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.896565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.896536 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.897421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.897398 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:34.951262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:34.951243 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzfv\" (UniqueName: \"kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv\") pod \"maas-api-revoke-sub-ae594979-bxs55\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:35.226610 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:35.226587 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:35.593003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:35.592982 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55"] Nov 02 22:26:35.594607 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:26:35.594580 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b90b0e5_d9e0_4aca_bf15_15a9e0167942.slice/crio-08f53b5513d49610d916cdd2aa1b069c8b46f84c74e1e5a66dc3ea97df05400d WatchSource:0}: Error finding container 08f53b5513d49610d916cdd2aa1b069c8b46f84c74e1e5a66dc3ea97df05400d: Status 404 returned error can't find the container with id 08f53b5513d49610d916cdd2aa1b069c8b46f84c74e1e5a66dc3ea97df05400d Nov 02 22:26:36.071036 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:36.071003 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" event={"ID":"9b90b0e5-d9e0-4aca-bf15-15a9e0167942","Type":"ContainerStarted","Data":"383266578cb52e42193376bab4f6d22cb04ea86e5ff7f5844a477daabbc3fa32"} Nov 02 22:26:36.071389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:36.071052 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" event={"ID":"9b90b0e5-d9e0-4aca-bf15-15a9e0167942","Type":"ContainerStarted","Data":"08f53b5513d49610d916cdd2aa1b069c8b46f84c74e1e5a66dc3ea97df05400d"} Nov 02 22:26:36.183958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:36.183914 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" podStartSLOduration=2.183900198 podStartE2EDuration="2.183900198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:36.183466317 +0000 UTC m=+3135.145016636" watchObservedRunningTime="2026-11-02 22:26:36.183900198 +0000 UTC m=+3135.145450516" Nov 02 22:26:37.077700 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:37.077665 4285 generic.go:356] "Generic (PLEG): container finished" podID="9b90b0e5-d9e0-4aca-bf15-15a9e0167942" containerID="383266578cb52e42193376bab4f6d22cb04ea86e5ff7f5844a477daabbc3fa32" exitCode=0 Nov 02 22:26:37.078101 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:37.077709 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" event={"ID":"9b90b0e5-d9e0-4aca-bf15-15a9e0167942","Type":"ContainerDied","Data":"383266578cb52e42193376bab4f6d22cb04ea86e5ff7f5844a477daabbc3fa32"} Nov 02 22:26:38.206034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.206005 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:38.224185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.224152 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca\") pod \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " Nov 02 22:26:38.224309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.224210 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv\" (UniqueName: \"kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv\") pod \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " Nov 02 22:26:38.224309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.224281 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca\") pod \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\" (UID: \"9b90b0e5-d9e0-4aca-bf15-15a9e0167942\") " Nov 02 22:26:38.224674 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.224645 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca" pod "9b90b0e5-d9e0-4aca-bf15-15a9e0167942" (UID: "9b90b0e5-d9e0-4aca-bf15-15a9e0167942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:38.226102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.226086 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv" pod "9b90b0e5-d9e0-4aca-bf15-15a9e0167942" (UID: "9b90b0e5-d9e0-4aca-bf15-15a9e0167942"). InnerVolumeSpecName "kube-api-access-6lzfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:38.226235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.226214 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca" pod "9b90b0e5-d9e0-4aca-bf15-15a9e0167942" (UID: "9b90b0e5-d9e0-4aca-bf15-15a9e0167942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:38.325409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.325379 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.325527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.325411 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lzfv\" (UniqueName: \"kubernetes.io/projected/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-kube-api-access-6lzfv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:26:38.325527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:38.325427 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b90b0e5-d9e0-4aca-bf15-15a9e0167942-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:26:39.087895 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:39.087863 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" Nov 02 22:26:39.088041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:39.087896 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55" event={"ID":"9b90b0e5-d9e0-4aca-bf15-15a9e0167942","Type":"ContainerDied","Data":"08f53b5513d49610d916cdd2aa1b069c8b46f84c74e1e5a66dc3ea97df05400d"} Nov 02 22:26:39.088041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:26:39.087944 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f53b5513d49610d916cdd2aa1b069c8b46f84c74e1e5a66dc3ea97df05400d" Nov 02 22:26:46.004896 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:26:46.004861 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:27:02.518586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.518550 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r"] Nov 02 22:27:02.520938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.519019 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9b90b0e5-d9e0-4aca-bf15-15a9e0167942" containerName="revoke-keys" Nov 02 22:27:02.520938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.519032 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b90b0e5-d9e0-4aca-bf15-15a9e0167942" containerName="revoke-keys" Nov 02 22:27:02.520938 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.519097 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b90b0e5-d9e0-4aca-bf15-15a9e0167942" containerName="revoke-keys" Nov 02 22:27:02.522101 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.522085 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.575919 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.575888 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:02.625491 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.625457 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:27:02.627726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.627702 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r"] Nov 02 22:27:02.671820 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.671792 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.671963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.671823 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.671963 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.671852 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwkv\" (UniqueName: \"kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.773461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.773381 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.773461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.773421 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.773671 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.773464 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwkv\" (UniqueName: \"kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.774058 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.774036 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.775653 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.775629 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.829514 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.829494 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwkv\" (UniqueName: \"kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv\") pod \"maas-api-revoke-sub-a4aef7d7-nk76r\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:02.834289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:02.834266 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:03.207384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:03.207358 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r"] Nov 02 22:27:03.208924 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:27:03.208898 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e7c846_c3ad_455a_a4d3_2330800637ee.slice/crio-0b6b8832a7c3079a1a21e72a6ed1ddea04f3f0bbf7d1b59ee9826fcfaec97b03 WatchSource:0}: Error finding container 0b6b8832a7c3079a1a21e72a6ed1ddea04f3f0bbf7d1b59ee9826fcfaec97b03: Status 404 returned error can't find the container with id 0b6b8832a7c3079a1a21e72a6ed1ddea04f3f0bbf7d1b59ee9826fcfaec97b03 Nov 02 22:27:03.225574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:03.225551 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" event={"ID":"41e7c846-c3ad-455a-a4d3-2330800637ee","Type":"ContainerStarted","Data":"0b6b8832a7c3079a1a21e72a6ed1ddea04f3f0bbf7d1b59ee9826fcfaec97b03"} Nov 02 22:27:04.231057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:04.231025 4285 generic.go:356] "Generic (PLEG): container finished" podID="41e7c846-c3ad-455a-a4d3-2330800637ee" containerID="efa4ee025b72fa3615298edd0219cff8cd34726f4577cd4752a23e9fa3df8d3a" exitCode=0 Nov 02 22:27:04.231469 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:04.231110 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" event={"ID":"41e7c846-c3ad-455a-a4d3-2330800637ee","Type":"ContainerDied","Data":"efa4ee025b72fa3615298edd0219cff8cd34726f4577cd4752a23e9fa3df8d3a"} Nov 02 22:27:05.360897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.360875 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:05.496321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.496259 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca\") pod \"41e7c846-c3ad-455a-a4d3-2330800637ee\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " Nov 02 22:27:05.496321 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.496296 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca\") pod \"41e7c846-c3ad-455a-a4d3-2330800637ee\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " Nov 02 22:27:05.496487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.496324 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv\" (UniqueName: \"kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv\") pod \"41e7c846-c3ad-455a-a4d3-2330800637ee\" (UID: \"41e7c846-c3ad-455a-a4d3-2330800637ee\") " Nov 02 22:27:05.496683 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.496654 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca" pod "41e7c846-c3ad-455a-a4d3-2330800637ee" (UID: "41e7c846-c3ad-455a-a4d3-2330800637ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:05.496808 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.496745 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.498535 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.498506 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv" pod "41e7c846-c3ad-455a-a4d3-2330800637ee" (UID: "41e7c846-c3ad-455a-a4d3-2330800637ee"). InnerVolumeSpecName "kube-api-access-xcwkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:05.498664 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.498539 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca" pod "41e7c846-c3ad-455a-a4d3-2330800637ee" (UID: "41e7c846-c3ad-455a-a4d3-2330800637ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:05.597979 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.597951 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e7c846-c3ad-455a-a4d3-2330800637ee-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:05.597979 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:05.597979 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwkv\" (UniqueName: \"kubernetes.io/projected/41e7c846-c3ad-455a-a4d3-2330800637ee-kube-api-access-xcwkv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:06.241336 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:06.241307 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" Nov 02 22:27:06.241489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:06.241346 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r" event={"ID":"41e7c846-c3ad-455a-a4d3-2330800637ee","Type":"ContainerDied","Data":"0b6b8832a7c3079a1a21e72a6ed1ddea04f3f0bbf7d1b59ee9826fcfaec97b03"} Nov 02 22:27:06.241489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:06.241387 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6b8832a7c3079a1a21e72a6ed1ddea04f3f0bbf7d1b59ee9826fcfaec97b03" Nov 02 22:27:16.004476 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:27:16.004432 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:27:21.928917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:21.928864 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:33.041955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.041917 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp"] Nov 02 22:27:33.042524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.042503 4285 cpu_manager.go:414] "containerMap: removing container" podUID="41e7c846-c3ad-455a-a4d3-2330800637ee" containerName="revoke-keys" Nov 02 22:27:33.042617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.042527 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e7c846-c3ad-455a-a4d3-2330800637ee" containerName="revoke-keys" Nov 02 22:27:33.042617 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.042608 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e7c846-c3ad-455a-a4d3-2330800637ee" containerName="revoke-keys" Nov 02 22:27:33.045905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.045885 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.066655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.066626 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.066744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.066680 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.066842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.066822 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pdk\" (UniqueName: \"kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.098165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.098144 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:33.147574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.147549 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:27:33.149853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.149830 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp"] Nov 02 22:27:33.168306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.168282 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pdk\" (UniqueName: \"kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.168401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.168329 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.168530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.168508 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.169179 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.169158 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.170795 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.170776 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.225041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.225019 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pdk\" (UniqueName: \"kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk\") pod \"maas-api-revoke-sub-79805ca9-6zpzp\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.358563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.358529 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:33.727878 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:33.727855 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp"] Nov 02 22:27:33.729269 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:27:33.729238 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63415894_08ee_40c8_94ac_05191ce39c04.slice/crio-71f7b809110b6c0a124590c392f4184a7bf568e317f6576285712f5954fbaffd WatchSource:0}: Error finding container 71f7b809110b6c0a124590c392f4184a7bf568e317f6576285712f5954fbaffd: Status 404 returned error can't find the container with id 71f7b809110b6c0a124590c392f4184a7bf568e317f6576285712f5954fbaffd Nov 02 22:27:34.414800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:34.414705 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" event={"ID":"63415894-08ee-40c8-94ac-05191ce39c04","Type":"ContainerStarted","Data":"00ca50f94c85321db7d14dcd522339ca8e6e74dc320ce80c23aae5ce8f868357"} Nov 02 22:27:34.414800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:34.414762 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" event={"ID":"63415894-08ee-40c8-94ac-05191ce39c04","Type":"ContainerStarted","Data":"71f7b809110b6c0a124590c392f4184a7bf568e317f6576285712f5954fbaffd"} Nov 02 22:27:34.528845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:34.528793 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" podStartSLOduration=2.528780566 podStartE2EDuration="2.528780566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:34.526344042 +0000 UTC m=+3193.487894360" watchObservedRunningTime="2026-11-02 22:27:34.528780566 +0000 UTC m=+3193.490330937" Nov 02 22:27:35.424799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:35.424764 4285 generic.go:356] "Generic (PLEG): container finished" podID="63415894-08ee-40c8-94ac-05191ce39c04" containerID="00ca50f94c85321db7d14dcd522339ca8e6e74dc320ce80c23aae5ce8f868357" exitCode=0 Nov 02 22:27:35.425203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:35.424847 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" event={"ID":"63415894-08ee-40c8-94ac-05191ce39c04","Type":"ContainerDied","Data":"00ca50f94c85321db7d14dcd522339ca8e6e74dc320ce80c23aae5ce8f868357"} Nov 02 22:27:36.554488 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.554466 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:36.603654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.603628 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk\" (UniqueName: \"kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk\") pod \"63415894-08ee-40c8-94ac-05191ce39c04\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " Nov 02 22:27:36.603795 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.603659 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca\") pod \"63415894-08ee-40c8-94ac-05191ce39c04\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " Nov 02 22:27:36.603795 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.603782 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca\") pod \"63415894-08ee-40c8-94ac-05191ce39c04\" (UID: \"63415894-08ee-40c8-94ac-05191ce39c04\") " Nov 02 22:27:36.604037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.604016 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca" pod "63415894-08ee-40c8-94ac-05191ce39c04" (UID: "63415894-08ee-40c8-94ac-05191ce39c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:36.605871 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.605838 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk" pod "63415894-08ee-40c8-94ac-05191ce39c04" (UID: "63415894-08ee-40c8-94ac-05191ce39c04"). InnerVolumeSpecName "kube-api-access-57pdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:36.605964 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.605874 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca" pod "63415894-08ee-40c8-94ac-05191ce39c04" (UID: "63415894-08ee-40c8-94ac-05191ce39c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:36.704900 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.704875 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pdk\" (UniqueName: \"kubernetes.io/projected/63415894-08ee-40c8-94ac-05191ce39c04-kube-api-access-57pdk\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:36.704900 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.704900 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:36.705062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:36.704910 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63415894-08ee-40c8-94ac-05191ce39c04-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:37.436452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:37.436381 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" Nov 02 22:27:37.436452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:37.436400 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp" event={"ID":"63415894-08ee-40c8-94ac-05191ce39c04","Type":"ContainerDied","Data":"71f7b809110b6c0a124590c392f4184a7bf568e317f6576285712f5954fbaffd"} Nov 02 22:27:37.436452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:37.436442 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f7b809110b6c0a124590c392f4184a7bf568e317f6576285712f5954fbaffd" Nov 02 22:27:46.004593 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:27:46.004551 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:27:56.831925 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.831888 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl"] Nov 02 22:27:56.832477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.832324 4285 cpu_manager.go:414] "containerMap: removing container" podUID="63415894-08ee-40c8-94ac-05191ce39c04" containerName="revoke-keys" Nov 02 22:27:56.832477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.832342 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63415894-08ee-40c8-94ac-05191ce39c04" containerName="revoke-keys" Nov 02 22:27:56.832477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.832443 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="63415894-08ee-40c8-94ac-05191ce39c04" containerName="revoke-keys" Nov 02 22:27:56.835756 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.835738 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:56.890579 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.890553 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:56.894220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.894195 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl"] Nov 02 22:27:56.920629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.920606 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:56.920780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.920644 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:56.920780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.920686 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhqb\" (UniqueName: \"kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:56.940044 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:56.940022 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:27:57.022095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.022071 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.022233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.022108 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.022233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.022163 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhqb\" (UniqueName: \"kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.022778 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.022754 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.024252 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.024233 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.078301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.078283 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhqb\" (UniqueName: \"kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb\") pod \"maas-api-revoke-sub-0b7e8edb-z9cnl\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.148540 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.148492 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:57.516139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.516098 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl"] Nov 02 22:27:57.517279 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:27:57.517250 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4135bc6_a5c1_4168_b50c_44f90ea3b6c5.slice/crio-66af4f5e575047a015893ab40e7dd7b7c17b77d990a93e0592ed62828fca60ca WatchSource:0}: Error finding container 66af4f5e575047a015893ab40e7dd7b7c17b77d990a93e0592ed62828fca60ca: Status 404 returned error can't find the container with id 66af4f5e575047a015893ab40e7dd7b7c17b77d990a93e0592ed62828fca60ca Nov 02 22:27:57.542900 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:57.542868 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" event={"ID":"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5","Type":"ContainerStarted","Data":"66af4f5e575047a015893ab40e7dd7b7c17b77d990a93e0592ed62828fca60ca"} Nov 02 22:27:58.549908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:58.549821 4285 generic.go:356] "Generic (PLEG): container finished" podID="d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" containerID="a0fc1cb6e75f9cdef76097988cdf06feb7e85ee5ca3097bf7725ede636a23370" exitCode=0 Nov 02 22:27:58.549908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:58.549862 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" event={"ID":"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5","Type":"ContainerDied","Data":"a0fc1cb6e75f9cdef76097988cdf06feb7e85ee5ca3097bf7725ede636a23370"} Nov 02 22:27:59.681001 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.680974 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:27:59.845454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.845387 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb\" (UniqueName: \"kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb\") pod \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " Nov 02 22:27:59.845592 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.845457 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca\") pod \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " Nov 02 22:27:59.845644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.845625 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca\") pod \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\" (UID: \"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5\") " Nov 02 22:27:59.845748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.845729 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca" pod "d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" (UID: "d4135bc6-a5c1-4168-b50c-44f90ea3b6c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:59.845957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.845939 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:59.847462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.847432 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca" pod "d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" (UID: "d4135bc6-a5c1-4168-b50c-44f90ea3b6c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:59.847561 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.847485 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb" pod "d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" (UID: "d4135bc6-a5c1-4168-b50c-44f90ea3b6c5"). InnerVolumeSpecName "kube-api-access-pfhqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:59.946792 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.946766 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:27:59.946792 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:27:59.946791 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhqb\" (UniqueName: \"kubernetes.io/projected/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5-kube-api-access-pfhqb\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:00.561019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:00.560988 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" Nov 02 22:28:00.561213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:00.561017 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl" event={"ID":"d4135bc6-a5c1-4168-b50c-44f90ea3b6c5","Type":"ContainerDied","Data":"66af4f5e575047a015893ab40e7dd7b7c17b77d990a93e0592ed62828fca60ca"} Nov 02 22:28:00.561213 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:00.561076 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66af4f5e575047a015893ab40e7dd7b7c17b77d990a93e0592ed62828fca60ca" Nov 02 22:28:08.157727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.157694 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4"] Nov 02 22:28:08.158210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.158094 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" containerName="revoke-keys" Nov 02 22:28:08.158210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.158106 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" containerName="revoke-keys" Nov 02 22:28:08.158210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.158199 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" containerName="revoke-keys" Nov 02 22:28:08.161400 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.161380 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.219315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.219285 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4"] Nov 02 22:28:08.265709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.265680 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:08.265876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.265731 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:28:08.323268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.323236 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gtr\" (UniqueName: \"kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.323455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.323429 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.323562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.323536 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.424572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.424514 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.424572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.424552 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.424775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.424604 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gtr\" (UniqueName: \"kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.425111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.425091 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.426790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.426769 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.481652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.481629 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gtr\" (UniqueName: \"kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr\") pod \"maas-api-revoke-sub-3d4156f1-qxxz4\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:08.775151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:08.775111 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:09.142246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:09.142221 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4"] Nov 02 22:28:09.144652 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:28:09.144621 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdcef9d_2601_4d86_b0dd_3ea2b48b7770.slice/crio-e214b8ae79b6475d6f6f5842bf5789e6c7cda08fac3b8b3ac2ee555d3ab0ff5e WatchSource:0}: Error finding container e214b8ae79b6475d6f6f5842bf5789e6c7cda08fac3b8b3ac2ee555d3ab0ff5e: Status 404 returned error can't find the container with id e214b8ae79b6475d6f6f5842bf5789e6c7cda08fac3b8b3ac2ee555d3ab0ff5e Nov 02 22:28:09.606066 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:09.606035 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" event={"ID":"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770","Type":"ContainerStarted","Data":"38082e780bab2472a90979798f57abfafeee148803692991432a57753d1a9cce"} Nov 02 22:28:09.606571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:09.606089 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" event={"ID":"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770","Type":"ContainerStarted","Data":"e214b8ae79b6475d6f6f5842bf5789e6c7cda08fac3b8b3ac2ee555d3ab0ff5e"} Nov 02 22:28:09.726918 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:09.726868 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" podStartSLOduration=1.726853342 podStartE2EDuration="1.726853342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:28:09.725798775 +0000 UTC m=+3228.687349093" watchObservedRunningTime="2026-11-02 22:28:09.726853342 +0000 UTC m=+3228.688403660" Nov 02 22:28:10.619150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:10.617970 4285 generic.go:356] "Generic (PLEG): container finished" podID="1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" containerID="38082e780bab2472a90979798f57abfafeee148803692991432a57753d1a9cce" exitCode=0 Nov 02 22:28:10.619150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:10.618089 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" event={"ID":"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770","Type":"ContainerDied","Data":"38082e780bab2472a90979798f57abfafeee148803692991432a57753d1a9cce"} Nov 02 22:28:11.754394 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.754370 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:11.786488 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.786458 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca\") pod \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " Nov 02 22:28:11.786655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.786500 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr\" (UniqueName: \"kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr\") pod \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " Nov 02 22:28:11.786655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.786526 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca\") pod \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\" (UID: \"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770\") " Nov 02 22:28:11.786868 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.786844 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca" pod "1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" (UID: "1cdcef9d-2601-4d86-b0dd-3ea2b48b7770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:11.788564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.788531 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca" pod "1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" (UID: "1cdcef9d-2601-4d86-b0dd-3ea2b48b7770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:11.788669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.788626 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr" pod "1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" (UID: "1cdcef9d-2601-4d86-b0dd-3ea2b48b7770"). InnerVolumeSpecName "kube-api-access-79gtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:11.887404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.887339 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gtr\" (UniqueName: \"kubernetes.io/projected/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-kube-api-access-79gtr\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:11.887404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.887364 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:11.887404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:11.887376 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:12.629616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:12.629587 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" Nov 02 22:28:12.629788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:12.629634 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4" event={"ID":"1cdcef9d-2601-4d86-b0dd-3ea2b48b7770","Type":"ContainerDied","Data":"e214b8ae79b6475d6f6f5842bf5789e6c7cda08fac3b8b3ac2ee555d3ab0ff5e"} Nov 02 22:28:12.629788 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:12.629679 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e214b8ae79b6475d6f6f5842bf5789e6c7cda08fac3b8b3ac2ee555d3ab0ff5e" Nov 02 22:28:16.004484 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:28:16.004446 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:28:20.435307 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:20.435262 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:23.053948 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:23.053910 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls"] Nov 02 22:28:23.102984 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:23.102955 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-txqls"] Nov 02 22:28:23.181811 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:23.181767 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2994c34d-c7fb-47f7-bc31-a583dc8473c6" path="/var/lib/kubelet/pods/2994c34d-c7fb-47f7-bc31-a583dc8473c6/volumes" Nov 02 22:28:29.199598 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.199569 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4"] Nov 02 22:28:29.248677 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.248650 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qxxz4"] Nov 02 22:28:29.317452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.317424 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf"] Nov 02 22:28:29.317793 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.317781 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" containerName="revoke-keys" Nov 02 22:28:29.317839 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.317795 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" containerName="revoke-keys" Nov 02 22:28:29.317881 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.317872 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" containerName="revoke-keys" Nov 02 22:28:29.321097 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.321082 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.342722 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.342698 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.342819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.342730 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwh6z\" (UniqueName: \"kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.342819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.342761 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.373800 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.373770 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:29.373950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.373937 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:28:29.379522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.379497 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf"] Nov 02 22:28:29.443675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.443649 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.443842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.443687 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwh6z\" (UniqueName: \"kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.443842 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.443723 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.444372 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.444351 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.445988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.445956 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.500042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.499988 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwh6z\" (UniqueName: \"kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z\") pod \"maas-api-revoke-sub-3d4156f1-lwbhf\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:29.633389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:29.633363 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:30.009357 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:30.009330 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf"] Nov 02 22:28:30.010745 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:28:30.010713 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1b4678_8440_474b_975d_b1d5a6361944.slice/crio-48a6d9653b4b8d5da03aae7bcab04803609f182abc5b2c37945be5a5bcdd4acb WatchSource:0}: Error finding container 48a6d9653b4b8d5da03aae7bcab04803609f182abc5b2c37945be5a5bcdd4acb: Status 404 returned error can't find the container with id 48a6d9653b4b8d5da03aae7bcab04803609f182abc5b2c37945be5a5bcdd4acb Nov 02 22:28:30.725569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:30.725531 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" event={"ID":"0f1b4678-8440-474b-975d-b1d5a6361944","Type":"ContainerStarted","Data":"0329aee5f1e70f7a27372cca7ce8d3f00fdb0a656b09f38580a30e18f02f8e64"} Nov 02 22:28:30.725915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:30.725600 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" event={"ID":"0f1b4678-8440-474b-975d-b1d5a6361944","Type":"ContainerStarted","Data":"48a6d9653b4b8d5da03aae7bcab04803609f182abc5b2c37945be5a5bcdd4acb"} Nov 02 22:28:30.840407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:30.840362 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" podStartSLOduration=1.840347738 podStartE2EDuration="1.840347738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:28:30.838459632 +0000 UTC m=+3249.800009951" watchObservedRunningTime="2026-11-02 22:28:30.840347738 +0000 UTC m=+3249.801898057" Nov 02 22:28:31.172652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:31.172580 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cdcef9d-2601-4d86-b0dd-3ea2b48b7770" path="/var/lib/kubelet/pods/1cdcef9d-2601-4d86-b0dd-3ea2b48b7770/volumes" Nov 02 22:28:31.732350 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:31.732321 4285 generic.go:356] "Generic (PLEG): container finished" podID="0f1b4678-8440-474b-975d-b1d5a6361944" containerID="0329aee5f1e70f7a27372cca7ce8d3f00fdb0a656b09f38580a30e18f02f8e64" exitCode=0 Nov 02 22:28:31.732744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:31.732412 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" event={"ID":"0f1b4678-8440-474b-975d-b1d5a6361944","Type":"ContainerDied","Data":"0329aee5f1e70f7a27372cca7ce8d3f00fdb0a656b09f38580a30e18f02f8e64"} Nov 02 22:28:32.876013 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.875990 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:32.975703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.975676 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z\" (UniqueName: \"kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z\") pod \"0f1b4678-8440-474b-975d-b1d5a6361944\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " Nov 02 22:28:32.975863 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.975750 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca\") pod \"0f1b4678-8440-474b-975d-b1d5a6361944\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " Nov 02 22:28:32.975863 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.975785 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca\") pod \"0f1b4678-8440-474b-975d-b1d5a6361944\" (UID: \"0f1b4678-8440-474b-975d-b1d5a6361944\") " Nov 02 22:28:32.976192 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.976164 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca" pod "0f1b4678-8440-474b-975d-b1d5a6361944" (UID: "0f1b4678-8440-474b-975d-b1d5a6361944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:32.977734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.977712 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca" pod "0f1b4678-8440-474b-975d-b1d5a6361944" (UID: "0f1b4678-8440-474b-975d-b1d5a6361944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:32.977818 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:32.977775 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z" pod "0f1b4678-8440-474b-975d-b1d5a6361944" (UID: "0f1b4678-8440-474b-975d-b1d5a6361944"). InnerVolumeSpecName "kube-api-access-hwh6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:33.077121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:33.077059 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:33.077121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:33.077086 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f1b4678-8440-474b-975d-b1d5a6361944-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:33.077121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:33.077097 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwh6z\" (UniqueName: \"kubernetes.io/projected/0f1b4678-8440-474b-975d-b1d5a6361944-kube-api-access-hwh6z\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:28:33.745289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:33.745255 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" event={"ID":"0f1b4678-8440-474b-975d-b1d5a6361944","Type":"ContainerDied","Data":"48a6d9653b4b8d5da03aae7bcab04803609f182abc5b2c37945be5a5bcdd4acb"} Nov 02 22:28:33.745289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:33.745283 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf" Nov 02 22:28:33.745487 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:28:33.745309 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a6d9653b4b8d5da03aae7bcab04803609f182abc5b2c37945be5a5bcdd4acb" Nov 02 22:28:46.003632 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:28:46.003597 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:29:03.735024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.734942 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx"] Nov 02 22:29:03.735560 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.735539 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0f1b4678-8440-474b-975d-b1d5a6361944" containerName="revoke-keys" Nov 02 22:29:03.735647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.735563 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1b4678-8440-474b-975d-b1d5a6361944" containerName="revoke-keys" Nov 02 22:29:03.735703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.735653 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1b4678-8440-474b-975d-b1d5a6361944" containerName="revoke-keys" Nov 02 22:29:03.738735 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.738709 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.796142 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.796098 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx"] Nov 02 22:29:03.841410 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.841384 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:03.841561 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.841497 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:29:03.848255 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.848235 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.848324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.848273 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdp2b\" (UniqueName: \"kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.848393 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.848375 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.949526 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.949502 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.949684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.949570 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.949684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.949605 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdp2b\" (UniqueName: \"kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.950190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.950165 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:03.951897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:03.951869 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:04.005726 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:04.005677 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdp2b\" (UniqueName: \"kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b\") pod \"maas-api-revoke-sub-e2fc9c5b-vhbpx\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:04.052395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:04.052376 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:04.425345 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:04.425322 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx"] Nov 02 22:29:04.427424 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:04.427397 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae5f21d_b0e1_4bc3_a0f0_87a599f32736.slice/crio-8809dc31e2414da0feed545343425af5e7a0e41b12c1380fd98975915541a5d5 WatchSource:0}: Error finding container 8809dc31e2414da0feed545343425af5e7a0e41b12c1380fd98975915541a5d5: Status 404 returned error can't find the container with id 8809dc31e2414da0feed545343425af5e7a0e41b12c1380fd98975915541a5d5 Nov 02 22:29:04.932937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:04.932899 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" event={"ID":"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736","Type":"ContainerStarted","Data":"787e38e2a6dcf02968de13ed956cbf3c978c6a6ae36a6248ebe313aa94b63f99"} Nov 02 22:29:04.933328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:04.932977 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" event={"ID":"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736","Type":"ContainerStarted","Data":"8809dc31e2414da0feed545343425af5e7a0e41b12c1380fd98975915541a5d5"} Nov 02 22:29:05.047436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:05.047379 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" podStartSLOduration=2.047366103 podStartE2EDuration="2.047366103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:05.045049479 +0000 UTC m=+3284.006599799" watchObservedRunningTime="2026-11-02 22:29:05.047366103 +0000 UTC m=+3284.008916421" Nov 02 22:29:05.939528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:05.939493 4285 generic.go:356] "Generic (PLEG): container finished" podID="9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" containerID="787e38e2a6dcf02968de13ed956cbf3c978c6a6ae36a6248ebe313aa94b63f99" exitCode=0 Nov 02 22:29:05.939528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:05.939531 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" event={"ID":"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736","Type":"ContainerDied","Data":"787e38e2a6dcf02968de13ed956cbf3c978c6a6ae36a6248ebe313aa94b63f99"} Nov 02 22:29:07.073606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.073582 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:07.179346 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.179318 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca\") pod \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " Nov 02 22:29:07.179480 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.179354 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca\") pod \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " Nov 02 22:29:07.179561 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.179549 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b\" (UniqueName: \"kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b\") pod \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\" (UID: \"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736\") " Nov 02 22:29:07.179622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.179600 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca" pod "9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" (UID: "9ae5f21d-b0e1-4bc3-a0f0-87a599f32736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:07.179724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.179711 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.181337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.181309 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca" pod "9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" (UID: "9ae5f21d-b0e1-4bc3-a0f0-87a599f32736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:07.181500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.181479 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b" pod "9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" (UID: "9ae5f21d-b0e1-4bc3-a0f0-87a599f32736"). InnerVolumeSpecName "kube-api-access-jdp2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:07.280348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.280282 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdp2b\" (UniqueName: \"kubernetes.io/projected/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-kube-api-access-jdp2b\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.280348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.280310 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.951338 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.951310 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" Nov 02 22:29:07.951531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.951348 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx" event={"ID":"9ae5f21d-b0e1-4bc3-a0f0-87a599f32736","Type":"ContainerDied","Data":"8809dc31e2414da0feed545343425af5e7a0e41b12c1380fd98975915541a5d5"} Nov 02 22:29:07.951531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:07.951395 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8809dc31e2414da0feed545343425af5e7a0e41b12c1380fd98975915541a5d5" Nov 02 22:29:14.468521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.468478 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp"] Nov 02 22:29:14.469005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.468856 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" containerName="revoke-keys" Nov 02 22:29:14.469005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.468867 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" containerName="revoke-keys" Nov 02 22:29:14.469005 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.468949 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" containerName="revoke-keys" Nov 02 22:29:14.472077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.472056 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.543589 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.543553 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqw48\" (UniqueName: \"kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.543776 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.543691 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.543835 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.543774 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.575415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.575383 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:14.575550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.575432 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:29:14.577822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.577804 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp"] Nov 02 22:29:14.645265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.645232 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.645411 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.645325 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqw48\" (UniqueName: \"kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.645411 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.645404 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.645993 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.645958 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.647602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.647576 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.701946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.701914 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqw48\" (UniqueName: \"kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48\") pod \"maas-api-revoke-sub-5a29efb3-rfvsp\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:14.785423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:14.785365 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:15.160092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:15.160070 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp"] Nov 02 22:29:15.161967 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:15.161929 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab7cbd2_3a71_4876_81ff_7a4eda94c851.slice/crio-d909a4163253348a3608cfa5e88a1cb8fcb2cd59009b072833e5855cfed6fd3d WatchSource:0}: Error finding container d909a4163253348a3608cfa5e88a1cb8fcb2cd59009b072833e5855cfed6fd3d: Status 404 returned error can't find the container with id d909a4163253348a3608cfa5e88a1cb8fcb2cd59009b072833e5855cfed6fd3d Nov 02 22:29:15.996039 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:15.996005 4285 generic.go:356] "Generic (PLEG): container finished" podID="4ab7cbd2-3a71-4876-81ff-7a4eda94c851" containerID="bb3d5b53c11afa54ae162aeae8ba6bf09544c212f78ae3bdde20e4884f15acea" exitCode=0 Nov 02 22:29:15.996483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:15.996109 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" event={"ID":"4ab7cbd2-3a71-4876-81ff-7a4eda94c851","Type":"ContainerDied","Data":"bb3d5b53c11afa54ae162aeae8ba6bf09544c212f78ae3bdde20e4884f15acea"} Nov 02 22:29:15.996483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:15.996214 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" event={"ID":"4ab7cbd2-3a71-4876-81ff-7a4eda94c851","Type":"ContainerStarted","Data":"d909a4163253348a3608cfa5e88a1cb8fcb2cd59009b072833e5855cfed6fd3d"} Nov 02 22:29:16.008177 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:29:16.008146 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:29:17.159483 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.159453 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:17.271214 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.271099 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48\" (UniqueName: \"kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48\") pod \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " Nov 02 22:29:17.271397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.271232 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca\") pod \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " Nov 02 22:29:17.271397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.271271 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca\") pod \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\" (UID: \"4ab7cbd2-3a71-4876-81ff-7a4eda94c851\") " Nov 02 22:29:17.271627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.271589 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca" pod "4ab7cbd2-3a71-4876-81ff-7a4eda94c851" (UID: "4ab7cbd2-3a71-4876-81ff-7a4eda94c851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:17.278240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.278208 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca" pod "4ab7cbd2-3a71-4876-81ff-7a4eda94c851" (UID: "4ab7cbd2-3a71-4876-81ff-7a4eda94c851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:17.278377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.278249 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48" pod "4ab7cbd2-3a71-4876-81ff-7a4eda94c851" (UID: "4ab7cbd2-3a71-4876-81ff-7a4eda94c851"). InnerVolumeSpecName "kube-api-access-zqw48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:17.372831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.372796 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqw48\" (UniqueName: \"kubernetes.io/projected/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-kube-api-access-zqw48\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:17.372831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.372833 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:17.373092 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:17.372853 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab7cbd2-3a71-4876-81ff-7a4eda94c851-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:18.026148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:18.023538 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" event={"ID":"4ab7cbd2-3a71-4876-81ff-7a4eda94c851","Type":"ContainerDied","Data":"d909a4163253348a3608cfa5e88a1cb8fcb2cd59009b072833e5855cfed6fd3d"} Nov 02 22:29:18.026148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:18.023607 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d909a4163253348a3608cfa5e88a1cb8fcb2cd59009b072833e5855cfed6fd3d" Nov 02 22:29:18.026148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:18.023681 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp" Nov 02 22:29:20.871469 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:20.871433 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:24.507886 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:24.507844 4285 scope.go:122] "RemoveContainer" containerID="86840346638c065fa3f796f0692e6c6e3c7760b5547c2aa91b1c3f6f252a7599" Nov 02 22:29:31.003995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.003961 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf"] Nov 02 22:29:31.053325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.053289 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwbhf"] Nov 02 22:29:31.055650 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.055621 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658"] Nov 02 22:29:31.057500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.057482 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-5g658"] Nov 02 22:29:31.126328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.126297 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz"] Nov 02 22:29:31.126950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.126931 4285 cpu_manager.go:414] "containerMap: removing container" podUID="4ab7cbd2-3a71-4876-81ff-7a4eda94c851" containerName="revoke-keys" Nov 02 22:29:31.126999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.126955 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab7cbd2-3a71-4876-81ff-7a4eda94c851" containerName="revoke-keys" Nov 02 22:29:31.127077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.127065 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab7cbd2-3a71-4876-81ff-7a4eda94c851" containerName="revoke-keys" Nov 02 22:29:31.130402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.130385 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.173220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.173186 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1b4678-8440-474b-975d-b1d5a6361944" path="/var/lib/kubelet/pods/0f1b4678-8440-474b-975d-b1d5a6361944/volumes" Nov 02 22:29:31.173645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.173630 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cb211d-5a41-4487-a5f9-3d8eb601c13e" path="/var/lib/kubelet/pods/54cb211d-5a41-4487-a5f9-3d8eb601c13e/volumes" Nov 02 22:29:31.183588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.183566 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:31.184093 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.184078 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:29:31.186407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.186385 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz"] Nov 02 22:29:31.200210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.200186 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.200324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.200262 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghp6b\" (UniqueName: \"kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.200395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.200369 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.301306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.301235 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.301306 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.301282 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.301467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.301356 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghp6b\" (UniqueName: \"kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.301997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.301978 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.303571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.303549 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.358057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.358030 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghp6b\" (UniqueName: \"kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b\") pod \"maas-api-revoke-sub-3d4156f1-fwzvz\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.443208 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.443179 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:31.811635 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:31.811610 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz"] Nov 02 22:29:31.813404 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:31.813368 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbcbca3_5506_4d7e_bc33_62e80aa87d2f.slice/crio-82233a07395dbde8c83059c067fe7dc3161626fe08a4ea7f099832e7af6ace3b WatchSource:0}: Error finding container 82233a07395dbde8c83059c067fe7dc3161626fe08a4ea7f099832e7af6ace3b: Status 404 returned error can't find the container with id 82233a07395dbde8c83059c067fe7dc3161626fe08a4ea7f099832e7af6ace3b Nov 02 22:29:32.115254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:32.115218 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" event={"ID":"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f","Type":"ContainerStarted","Data":"82233a07395dbde8c83059c067fe7dc3161626fe08a4ea7f099832e7af6ace3b"} Nov 02 22:29:33.121076 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:33.121045 4285 generic.go:356] "Generic (PLEG): container finished" podID="9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" containerID="0e78f0246c5490d61139ae1f46639dee57abce92b46f6c1f0ba577ce8923dbef" exitCode=0 Nov 02 22:29:33.121425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:33.121139 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" event={"ID":"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f","Type":"ContainerDied","Data":"0e78f0246c5490d61139ae1f46639dee57abce92b46f6c1f0ba577ce8923dbef"} Nov 02 22:29:34.248682 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.248653 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:34.329484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.329452 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca\") pod \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " Nov 02 22:29:34.329625 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.329497 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b\" (UniqueName: \"kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b\") pod \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " Nov 02 22:29:34.329625 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.329608 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca\") pod \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\" (UID: \"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f\") " Nov 02 22:29:34.329999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.329967 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca" pod "9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" (UID: "9fbcbca3-5506-4d7e-bc33-62e80aa87d2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:34.331630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.331598 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca" pod "9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" (UID: "9fbcbca3-5506-4d7e-bc33-62e80aa87d2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:34.331727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.331663 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b" pod "9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" (UID: "9fbcbca3-5506-4d7e-bc33-62e80aa87d2f"). InnerVolumeSpecName "kube-api-access-ghp6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:34.430742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.430675 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:34.430742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.430699 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghp6b\" (UniqueName: \"kubernetes.io/projected/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-kube-api-access-ghp6b\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:34.430742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:34.430711 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.133496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:35.133466 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" Nov 02 22:29:35.133661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:35.133496 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz" event={"ID":"9fbcbca3-5506-4d7e-bc33-62e80aa87d2f","Type":"ContainerDied","Data":"82233a07395dbde8c83059c067fe7dc3161626fe08a4ea7f099832e7af6ace3b"} Nov 02 22:29:35.133661 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:35.133542 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82233a07395dbde8c83059c067fe7dc3161626fe08a4ea7f099832e7af6ace3b" Nov 02 22:29:46.004917 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:29:46.004881 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:29:47.233161 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.233112 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:47.737140 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.737101 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr"] Nov 02 22:29:47.737559 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.737544 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" containerName="revoke-keys" Nov 02 22:29:47.737620 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.737561 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" containerName="revoke-keys" Nov 02 22:29:47.737657 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.737636 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" containerName="revoke-keys" Nov 02 22:29:47.740973 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.740957 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842492 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8lv\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842586 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-53bfd6-openshift-default-dockercfg-hgpw4\"" Nov 02 22:29:47.842715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842612 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842694 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842754 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842803 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842836 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.842912 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842892 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.843288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842919 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.843288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.842951 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.844482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.844457 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr"] Nov 02 22:29:47.864273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.864251 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:47.944432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944403 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944466 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944510 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944547 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944599 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944616 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944645 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944683 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8lv\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.944758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944740 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.945341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.944989 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.945341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.945187 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.945576 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.945553 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.945679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.945632 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.945746 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.945694 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.947489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.947468 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:47.947784 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:47.947764 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:48.002518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.002400 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:48.011914 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.011882 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8lv\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv\") pod \"e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:48.052931 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.052904 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:48.073887 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.073848 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:48.477915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.477886 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr"] Nov 02 22:29:48.516902 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:48.516878 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5dfeb4_858b_4e30_a69d_cec9d22f12ed.slice/crio-09d9d75741b98085e05cb22079b252a5f2370d72a985ca1aa9a47b3b881f7d56 WatchSource:0}: Error finding container 09d9d75741b98085e05cb22079b252a5f2370d72a985ca1aa9a47b3b881f7d56: Status 404 returned error can't find the container with id 09d9d75741b98085e05cb22079b252a5f2370d72a985ca1aa9a47b3b881f7d56 Nov 02 22:29:48.519031 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.518999 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:48.519111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.519061 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:48.519111 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:48.519091 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:49.203921 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:49.203888 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" event={"ID":"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed","Type":"ContainerStarted","Data":"1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf"} Nov 02 22:29:49.204088 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:49.203939 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" event={"ID":"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed","Type":"ContainerStarted","Data":"09d9d75741b98085e05cb22079b252a5f2370d72a985ca1aa9a47b3b881f7d56"} Nov 02 22:29:49.321162 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:49.321102 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podStartSLOduration=2.321088561 podStartE2EDuration="2.321088561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:49.319091141 +0000 UTC m=+3328.280641460" watchObservedRunningTime="2026-11-02 22:29:49.321088561 +0000 UTC m=+3328.282638938" Nov 02 22:29:50.053996 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:50.053969 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:50.059590 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:50.059566 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:50.209168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:50.209120 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:50.209898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:50.209875 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:29:52.206599 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:52.206562 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:52.473854 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:52.473751 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:52.730515 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:52.730449 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:55.448571 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.448535 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:55.611034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.610998 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:55.827531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.827456 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv"] Nov 02 22:29:55.830916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.830901 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:55.884968 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.884943 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-53bfd6\"" Nov 02 22:29:55.922416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.922383 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:55.922539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.922492 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpsb\" (UniqueName: \"kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb\") pod \"payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:55.936808 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.936786 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv"] Nov 02 22:29:55.991348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.990118 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l"] Nov 02 22:29:55.995537 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:55.995512 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.023575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.023549 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpsb\" (UniqueName: \"kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb\") pod \"payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:56.023704 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.023632 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:56.024245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.024226 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:56.047750 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.047724 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-78c94ddf-dockercfg-j7hk8\"" Nov 02 22:29:56.050568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.050544 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l"] Nov 02 22:29:56.081784 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.081726 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpsb\" (UniqueName: \"kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb\") pod \"payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:56.124639 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.124614 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zbj\" (UniqueName: \"kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj\") pod \"payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.124727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.124665 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.143558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.143535 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:56.225310 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.225279 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zbj\" (UniqueName: \"kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj\") pod \"payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.225454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.225337 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.225942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.225924 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.281455 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.281432 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zbj\" (UniqueName: \"kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj\") pod \"payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.308274 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.308248 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:56.511565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.511540 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv"] Nov 02 22:29:56.512764 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:56.512736 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879f512b_285e_4719_a70d_9fc95a414d1a.slice/crio-32cb0867dca8b1e2d8e0ec2d1a7eba98fbc8aba52872ad2c0d16dd9801a5b8bd WatchSource:0}: Error finding container 32cb0867dca8b1e2d8e0ec2d1a7eba98fbc8aba52872ad2c0d16dd9801a5b8bd: Status 404 returned error can't find the container with id 32cb0867dca8b1e2d8e0ec2d1a7eba98fbc8aba52872ad2c0d16dd9801a5b8bd Nov 02 22:29:56.676439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:56.676413 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l"] Nov 02 22:29:56.677000 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:56.676978 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc51def_3ffd_42be_9d56_2ca149cb4616.slice/crio-d0fead7dae7a37bc18c38f3096113e33385ac1ddc7bcff60b5dba1f58fff2d6c WatchSource:0}: Error finding container d0fead7dae7a37bc18c38f3096113e33385ac1ddc7bcff60b5dba1f58fff2d6c: Status 404 returned error can't find the container with id d0fead7dae7a37bc18c38f3096113e33385ac1ddc7bcff60b5dba1f58fff2d6c Nov 02 22:29:57.244767 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.244737 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" event={"ID":"879f512b-285e-4719-a70d-9fc95a414d1a","Type":"ContainerStarted","Data":"f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2"} Nov 02 22:29:57.244946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.244785 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" event={"ID":"879f512b-285e-4719-a70d-9fc95a414d1a","Type":"ContainerStarted","Data":"32cb0867dca8b1e2d8e0ec2d1a7eba98fbc8aba52872ad2c0d16dd9801a5b8bd"} Nov 02 22:29:57.244946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.244811 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:29:57.246314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.246291 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" event={"ID":"bdc51def-3ffd-42be-9d56-2ca149cb4616","Type":"ContainerStarted","Data":"366d657f25141c6683b99c57ae64e3fe0818ddb30225f79293670b4e76f15853"} Nov 02 22:29:57.246367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.246334 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" event={"ID":"bdc51def-3ffd-42be-9d56-2ca149cb4616","Type":"ContainerStarted","Data":"d0fead7dae7a37bc18c38f3096113e33385ac1ddc7bcff60b5dba1f58fff2d6c"} Nov 02 22:29:57.246407 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.246399 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:29:57.359575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.359533 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" podStartSLOduration=2.359521999 podStartE2EDuration="2.359521999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:57.355967243 +0000 UTC m=+3336.317517563" watchObservedRunningTime="2026-11-02 22:29:57.359521999 +0000 UTC m=+3336.321072317" Nov 02 22:29:57.492258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:57.492195 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" podStartSLOduration=2.492173589 podStartE2EDuration="2.492173589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:57.478608599 +0000 UTC m=+3336.440158915" watchObservedRunningTime="2026-11-02 22:29:57.492173589 +0000 UTC m=+3336.453723909" Nov 02 22:29:58.130368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.130336 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm"] Nov 02 22:29:58.133958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.133939 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.235751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.235686 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-53bfd6-serving-cert\"" Nov 02 22:29:58.235751 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.235724 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cbb6p\"" Nov 02 22:29:58.235985 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.235778 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:29:58.238630 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.238602 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm"] Nov 02 22:29:58.245361 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.245333 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.245506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.245383 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.245579 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.245507 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk8l\" (UniqueName: \"kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.245642 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.245608 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.346877 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.346846 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk8l\" (UniqueName: \"kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.347050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.346946 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.347050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.346975 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.347050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.347044 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.347298 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:29:58.347082 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-serving-cert: secret "maas-api-e2e-worker-single-main-53bfd6-serving-cert" not found Nov 02 22:29:58.347298 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:29:58.347270 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls podName:ec652347-104d-4b40-b2d1-9601e583ba79 nodeName:}" failed. No retries permitted until 2026-11-02 22:29:58.847244465 +0000 UTC m=+3337.808794776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls") pod "maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" (UID: "ec652347-104d-4b40-b2d1-9601e583ba79") : secret "maas-api-e2e-worker-single-main-53bfd6-serving-cert" not found Nov 02 22:29:58.347670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.347640 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.349389 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.349366 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.365496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.365469 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:58.404176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.404076 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk8l\" (UniqueName: \"kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.523169 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.523116 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:58.679958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.679877 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:58.852886 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.852857 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:58.855060 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:58.855035 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:59.047518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:59.047484 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:29:59.418157 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:59.418117 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm"] Nov 02 22:29:59.419242 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:29:59.419216 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec652347_104d_4b40_b2d1_9601e583ba79.slice/crio-37794650619b485706a931e4b378922b255c09136399cf1dea8d41a171e83f4b WatchSource:0}: Error finding container 37794650619b485706a931e4b378922b255c09136399cf1dea8d41a171e83f4b: Status 404 returned error can't find the container with id 37794650619b485706a931e4b378922b255c09136399cf1dea8d41a171e83f4b Nov 02 22:29:59.420598 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:29:59.420581 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:30:00.262599 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:00.262566 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" event={"ID":"ec652347-104d-4b40-b2d1-9601e583ba79","Type":"ContainerStarted","Data":"37794650619b485706a931e4b378922b255c09136399cf1dea8d41a171e83f4b"} Nov 02 22:30:01.269696 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:01.269661 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" event={"ID":"ec652347-104d-4b40-b2d1-9601e583ba79","Type":"ContainerStarted","Data":"fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870"} Nov 02 22:30:01.270173 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:01.269817 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:30:01.386207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:01.386160 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" podStartSLOduration=2.500684428 podStartE2EDuration="3.386146892s" totalImagesPullingTime="885.462464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:01.383666654 +0000 UTC m=+3340.345216973" watchObservedRunningTime="2026-11-02 22:30:01.386146892 +0000 UTC m=+3340.347697210" Nov 02 22:30:03.044670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:03.044636 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7"] Nov 02 22:30:03.094463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:03.094430 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zk4c7"] Nov 02 22:30:03.171734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:03.171707 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a6e76b-06e1-4590-8d4e-33f66724f2ea" path="/var/lib/kubelet/pods/38a6e76b-06e1-4590-8d4e-33f66724f2ea/volumes" Nov 02 22:30:03.212006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:03.211977 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:07.280420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:07.280393 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:30:08.027672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:08.027636 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:08.255206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:08.255176 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:30:08.255404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:08.255244 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:30:12.719937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:12.719894 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:15.048687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.048653 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh"] Nov 02 22:30:15.097746 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.097719 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h9tjh"] Nov 02 22:30:15.172454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.172426 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df34952f-37c4-4aff-a711-14614a92ef5c" path="/var/lib/kubelet/pods/df34952f-37c4-4aff-a711-14614a92ef5c/volumes" Nov 02 22:30:15.373901 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.373821 4285 generic.go:356] "Generic (PLEG): container finished" podID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerID="1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf" exitCode=0 Nov 02 22:30:15.374043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.373896 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" event={"ID":"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed","Type":"ContainerDied","Data":"1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf"} Nov 02 22:30:15.374337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.374321 4285 scope.go:122] "RemoveContainer" containerID="1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf" Nov 02 22:30:15.375468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.375432 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:30:15.375579 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.375520 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:30:15.375645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:15.375578 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:30:16.004691 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:30:16.004660 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:30:16.381004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:16.380920 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" event={"ID":"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed","Type":"ContainerStarted","Data":"335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847"} Nov 02 22:30:17.039421 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.039384 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.053417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.053389 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:30:17.054360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.054335 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:30:17.054453 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.054380 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 22:30:17.634194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.634157 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.795684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.795648 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.957668 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:17.957642 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:18.053764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:18.053729 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:30:18.053942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:18.053792 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 22:30:19.053702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:19.053671 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:30:19.054219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:19.053722 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 22:30:20.053369 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:20.053332 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:30:20.053547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:20.053384 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 22:30:20.134829 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:20.134779 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:20.209847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:20.209815 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:30:21.054274 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:21.054233 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:30:21.054463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:21.054288 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 02 22:30:21.087824 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:21.087790 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:22.057711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:22.057672 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:30:22.058569 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:22.058551 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:30:22.521714 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:22.521683 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:27.214114 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:27.214083 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:31.913673 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:31.913641 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:36.705007 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:36.704916 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:41.428309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:41.428273 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:46.004796 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:30:46.004758 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:30:46.244258 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:46.244229 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:51.023688 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:51.023646 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:53.044768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:53.044735 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l"] Nov 02 22:30:53.093757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:53.093723 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zzm5l"] Nov 02 22:30:53.172917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:53.172879 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50d6d67-233d-44a7-be9c-7d42bf29ba91" path="/var/lib/kubelet/pods/a50d6d67-233d-44a7-be9c-7d42bf29ba91/volumes" Nov 02 22:30:55.830647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:30:55.830612 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:00.503063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:00.503027 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:04.044451 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:04.044419 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5"] Nov 02 22:31:04.094167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:04.094118 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rjxz5"] Nov 02 22:31:05.172264 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:05.172227 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8294d0b-2444-4a8b-993f-3e104e443c81" path="/var/lib/kubelet/pods/e8294d0b-2444-4a8b-993f-3e104e443c81/volumes" Nov 02 22:31:05.215399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:05.215372 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:09.909413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:09.909375 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:14.698647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:14.698613 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:16.004154 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:31:16.004108 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:31:16.835689 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:16.835658 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:19.754669 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:19.754620 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:21.438034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:21.438002 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:21.862664 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:21.862630 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:24.539450 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:24.539420 4285 scope.go:122] "RemoveContainer" containerID="a537965a00ec293eb9d9b578f856da90952c2d426b06e8c88c031e74ef6f3f4c" Nov 02 22:31:24.547627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:24.547607 4285 scope.go:122] "RemoveContainer" containerID="eb77b94904715f3dab00aa3436022f48b41be2e61a6a3a6e64512ac85a6b1453" Nov 02 22:31:24.555206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:24.555182 4285 scope.go:122] "RemoveContainer" containerID="d1a4c14a15bf2787a9eb7debf24ccac2c7bd6ad9419ec79f24db6f322c38fe51" Nov 02 22:31:24.700935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:24.700904 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:28.754347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:28.754311 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l"] Nov 02 22:31:28.859899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:28.859866 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv"] Nov 02 22:31:29.123334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:29.123247 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:29.269248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:29.269218 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:29.433924 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:29.433797 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:33.411884 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:33.411851 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm"] Nov 02 22:31:33.412301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:33.412032 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" podUID="ec652347-104d-4b40-b2d1-9601e583ba79" containerName="maas-api" containerID="cri-o://fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870" gracePeriod=30 Nov 02 22:31:33.531715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:33.531677 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:33.754872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:33.754808 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerName="payload-processing" containerID="cri-o://366d657f25141c6683b99c57ae64e3fe0818ddb30225f79293670b4e76f15853" gracePeriod=25 Nov 02 22:31:33.861231 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:33.861182 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" containerName="payload-pre-processing" containerID="cri-o://f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2" gracePeriod=25 Nov 02 22:31:34.650182 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.650156 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:31:34.786500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.786416 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca\") pod \"ec652347-104d-4b40-b2d1-9601e583ba79\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " Nov 02 22:31:34.786500 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.786448 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\") pod \"ec652347-104d-4b40-b2d1-9601e583ba79\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " Nov 02 22:31:34.786737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.786528 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs\") pod \"ec652347-104d-4b40-b2d1-9601e583ba79\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " Nov 02 22:31:34.786737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.786595 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l\" (UniqueName: \"kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l\") pod \"ec652347-104d-4b40-b2d1-9601e583ba79\" (UID: \"ec652347-104d-4b40-b2d1-9601e583ba79\") " Nov 02 22:31:34.786860 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.786807 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca" pod "ec652347-104d-4b40-b2d1-9601e583ba79" (UID: "ec652347-104d-4b40-b2d1-9601e583ba79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:34.788798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.788769 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs" pod "ec652347-104d-4b40-b2d1-9601e583ba79" (UID: "ec652347-104d-4b40-b2d1-9601e583ba79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:34.788914 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.788860 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l" pod "ec652347-104d-4b40-b2d1-9601e583ba79" (UID: "ec652347-104d-4b40-b2d1-9601e583ba79"). InnerVolumeSpecName "kube-api-access-5tk8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:34.788955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.788915 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls" pod "ec652347-104d-4b40-b2d1-9601e583ba79" (UID: "ec652347-104d-4b40-b2d1-9601e583ba79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:34.817402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.817372 4285 generic.go:356] "Generic (PLEG): container finished" podID="ec652347-104d-4b40-b2d1-9601e583ba79" containerID="fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870" exitCode=0 Nov 02 22:31:34.817535 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.817481 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" Nov 02 22:31:34.817535 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.817497 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" event={"ID":"ec652347-104d-4b40-b2d1-9601e583ba79","Type":"ContainerDied","Data":"fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870"} Nov 02 22:31:34.817649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.817555 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm" event={"ID":"ec652347-104d-4b40-b2d1-9601e583ba79","Type":"ContainerDied","Data":"37794650619b485706a931e4b378922b255c09136399cf1dea8d41a171e83f4b"} Nov 02 22:31:34.817649 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.817588 4285 scope.go:122] "RemoveContainer" containerID="fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870" Nov 02 22:31:34.826527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.826508 4285 scope.go:122] "RemoveContainer" containerID="fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870" Nov 02 22:31:34.826789 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:31:34.826770 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870\": container with ID starting with fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870 not found: ID does not exist" containerID="fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870" Nov 02 22:31:34.826865 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.826800 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870"} err="failed to get container status \"fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870\": rpc error: code = NotFound desc = could not find container \"fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870\": container with ID starting with fba4909ff5ff96cf250840803e9c8050b33df37eeff86ba037ffbda33deab870 not found: ID does not exist" Nov 02 22:31:34.887695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.887668 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tk8l\" (UniqueName: \"kubernetes.io/projected/ec652347-104d-4b40-b2d1-9601e583ba79-kube-api-access-5tk8l\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:34.887695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.887695 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec652347-104d-4b40-b2d1-9601e583ba79-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:34.887922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.887711 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:34.887922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.887722 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec652347-104d-4b40-b2d1-9601e583ba79-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:34.979806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.979777 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm"] Nov 02 22:31:34.985563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:34.985538 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-53bfd6-7fbdfc7f6-6cxwm"] Nov 02 22:31:35.074779 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:35.074714 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:35.172469 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:35.172443 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec652347-104d-4b40-b2d1-9601e583ba79" path="/var/lib/kubelet/pods/ec652347-104d-4b40-b2d1-9601e583ba79/volumes" Nov 02 22:31:35.390345 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:35.390255 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:36.087700 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:36.087666 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:38.253240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:38.253201 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:38.253641 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:38.253194 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:39.046638 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:39.046607 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55"] Nov 02 22:31:39.095937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:39.095910 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bxs55"] Nov 02 22:31:39.172078 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:39.172049 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b90b0e5-d9e0-4aca-bf15-15a9e0167942" path="/var/lib/kubelet/pods/9b90b0e5-d9e0-4aca-bf15-15a9e0167942/volumes" Nov 02 22:31:41.810556 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:41.810523 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:41.970789 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:41.970755 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:42.107947 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:42.107874 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr"] Nov 02 22:31:42.108093 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:42.108075 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" containerID="cri-o://335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847" gracePeriod=30 Nov 02 22:31:42.136843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:42.136815 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:43.581946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.581909 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz"] Nov 02 22:31:43.631552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.631523 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fwzvz"] Nov 02 22:31:43.704303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.704278 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84"] Nov 02 22:31:43.704665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.704653 4285 cpu_manager.go:414] "containerMap: removing container" podUID="ec652347-104d-4b40-b2d1-9601e583ba79" containerName="maas-api" Nov 02 22:31:43.704709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.704666 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec652347-104d-4b40-b2d1-9601e583ba79" containerName="maas-api" Nov 02 22:31:43.704758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.704750 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec652347-104d-4b40-b2d1-9601e583ba79" containerName="maas-api" Nov 02 22:31:43.710930 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.710910 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.763165 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.763118 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:43.763276 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.763223 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:31:43.764272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.764251 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84"] Nov 02 22:31:43.770702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.770680 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pb28\" (UniqueName: \"kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.770814 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.770735 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.770814 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.770764 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.871740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.871660 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pb28\" (UniqueName: \"kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.871740 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.871719 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.871926 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.871756 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.872314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.872290 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.873961 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.873941 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:43.928468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:43.928444 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pb28\" (UniqueName: \"kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28\") pod \"maas-api-revoke-sub-3d4156f1-tcx84\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:44.023683 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:44.023657 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:44.389550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:44.389525 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84"] Nov 02 22:31:44.391145 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:31:44.391095 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8fdaae_67a2_47b9_9581_061212594a97.slice/crio-526561f0cd3ced9a8493604dd281c7d86cc89629e0562d0f26d70a1f400103b3 WatchSource:0}: Error finding container 526561f0cd3ced9a8493604dd281c7d86cc89629e0562d0f26d70a1f400103b3: Status 404 returned error can't find the container with id 526561f0cd3ced9a8493604dd281c7d86cc89629e0562d0f26d70a1f400103b3 Nov 02 22:31:44.885018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:44.884985 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" event={"ID":"4c8fdaae-67a2-47b9-9581-061212594a97","Type":"ContainerStarted","Data":"1eaf5d840a910dbc1e64204e896af8f6c9962dd47733726d0ceeac04bc0172af"} Nov 02 22:31:44.885387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:44.885040 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" event={"ID":"4c8fdaae-67a2-47b9-9581-061212594a97","Type":"ContainerStarted","Data":"526561f0cd3ced9a8493604dd281c7d86cc89629e0562d0f26d70a1f400103b3"} Nov 02 22:31:44.997754 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:44.997711 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" podStartSLOduration=1.997698163 podStartE2EDuration="1.997698163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:44.997642683 +0000 UTC m=+3443.959193000" watchObservedRunningTime="2026-11-02 22:31:44.997698163 +0000 UTC m=+3443.959248482" Nov 02 22:31:45.172479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:45.172448 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbcbca3-5506-4d7e-bc33-62e80aa87d2f" path="/var/lib/kubelet/pods/9fbcbca3-5506-4d7e-bc33-62e80aa87d2f/volumes" Nov 02 22:31:45.894757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:45.894719 4285 generic.go:356] "Generic (PLEG): container finished" podID="4c8fdaae-67a2-47b9-9581-061212594a97" containerID="1eaf5d840a910dbc1e64204e896af8f6c9962dd47733726d0ceeac04bc0172af" exitCode=0 Nov 02 22:31:45.895198 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:45.894760 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" event={"ID":"4c8fdaae-67a2-47b9-9581-061212594a97","Type":"ContainerDied","Data":"1eaf5d840a910dbc1e64204e896af8f6c9962dd47733726d0ceeac04bc0172af"} Nov 02 22:31:46.004614 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:31:46.004581 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:31:47.032401 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.032377 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:47.100439 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.100410 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca\") pod \"4c8fdaae-67a2-47b9-9581-061212594a97\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " Nov 02 22:31:47.100553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.100492 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca\") pod \"4c8fdaae-67a2-47b9-9581-061212594a97\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " Nov 02 22:31:47.100634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.100618 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28\" (UniqueName: \"kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28\") pod \"4c8fdaae-67a2-47b9-9581-061212594a97\" (UID: \"4c8fdaae-67a2-47b9-9581-061212594a97\") " Nov 02 22:31:47.100780 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.100761 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca" pod "4c8fdaae-67a2-47b9-9581-061212594a97" (UID: "4c8fdaae-67a2-47b9-9581-061212594a97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:47.100923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.100907 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.102402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.102381 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca" pod "4c8fdaae-67a2-47b9-9581-061212594a97" (UID: "4c8fdaae-67a2-47b9-9581-061212594a97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:47.102537 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.102513 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28" pod "4c8fdaae-67a2-47b9-9581-061212594a97" (UID: "4c8fdaae-67a2-47b9-9581-061212594a97"). InnerVolumeSpecName "kube-api-access-9pb28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:47.202191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.202166 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pb28\" (UniqueName: \"kubernetes.io/projected/4c8fdaae-67a2-47b9-9581-061212594a97-kube-api-access-9pb28\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.202191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.202191 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8fdaae-67a2-47b9-9581-061212594a97-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.338899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.338874 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:31:47.403342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403315 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403372 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403403 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403454 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403445 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403470 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403505 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403566 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403561 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:47.403621 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403616 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403633 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:47.403927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403662 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv\") pod \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\" (UID: \"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed\") " Nov 02 22:31:47.403927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403723 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:47.403927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403885 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:47.403927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403889 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:47.404210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403942 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.404210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403964 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.404210 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.403977 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.405651 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.405610 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:47.405753 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.405686 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:31:47.405753 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.405708 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:47.405753 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.405721 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv" pod "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" (UID: "9f5dfeb4-858b-4e30-a69d-cec9d22f12ed"). InnerVolumeSpecName "kube-api-access-9f8lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:47.511378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.505471 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.511378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.505506 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.511378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.505520 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.511378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.505536 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.511378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.505554 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.511378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.505568 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8lv\" (UniqueName: \"kubernetes.io/projected/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed-kube-api-access-9f8lv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:47.907279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.907201 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" Nov 02 22:31:47.907279 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.907226 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84" event={"ID":"4c8fdaae-67a2-47b9-9581-061212594a97","Type":"ContainerDied","Data":"526561f0cd3ced9a8493604dd281c7d86cc89629e0562d0f26d70a1f400103b3"} Nov 02 22:31:47.907498 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.907296 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="526561f0cd3ced9a8493604dd281c7d86cc89629e0562d0f26d70a1f400103b3" Nov 02 22:31:47.908913 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.908886 4285 generic.go:356] "Generic (PLEG): container finished" podID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerID="335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847" exitCode=0 Nov 02 22:31:47.909041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.908947 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" Nov 02 22:31:47.909041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.908976 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" event={"ID":"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed","Type":"ContainerDied","Data":"335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847"} Nov 02 22:31:47.909041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.909035 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr" event={"ID":"9f5dfeb4-858b-4e30-a69d-cec9d22f12ed","Type":"ContainerDied","Data":"09d9d75741b98085e05cb22079b252a5f2370d72a985ca1aa9a47b3b881f7d56"} Nov 02 22:31:47.909256 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.909067 4285 scope.go:122] "RemoveContainer" containerID="335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847" Nov 02 22:31:47.919530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.919514 4285 scope.go:122] "RemoveContainer" containerID="1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf" Nov 02 22:31:47.927254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.927209 4285 scope.go:122] "RemoveContainer" containerID="335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847" Nov 02 22:31:47.927485 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:31:47.927463 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847\": container with ID starting with 335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847 not found: ID does not exist" containerID="335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847" Nov 02 22:31:47.927553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.927492 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847"} err="failed to get container status \"335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847\": rpc error: code = NotFound desc = could not find container \"335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847\": container with ID starting with 335077b02d6d401b78ce3f7ec5d413e2df37eabda0bf5d927c5095ce6fea6847 not found: ID does not exist" Nov 02 22:31:47.927553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.927512 4285 scope.go:122] "RemoveContainer" containerID="1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf" Nov 02 22:31:47.927760 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:31:47.927744 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf\": container with ID starting with 1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf not found: ID does not exist" containerID="1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf" Nov 02 22:31:47.927822 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:47.927766 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf"} err="failed to get container status \"1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf\": rpc error: code = NotFound desc = could not find container \"1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf\": container with ID starting with 1b3287d68ad911dfd45293f4323b0cd260271d7469db1961d4a33ed80c8f62bf not found: ID does not exist" Nov 02 22:31:48.072426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:48.072394 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr"] Nov 02 22:31:48.083098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:48.083072 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-53bfd6-openshift-default-6d975f6db56kkbr"] Nov 02 22:31:48.253688 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:48.253653 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:48.253688 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:48.253674 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:49.172533 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:49.172503 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" path="/var/lib/kubelet/pods/9f5dfeb4-858b-4e30-a69d-cec9d22f12ed/volumes" Nov 02 22:31:53.942417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:53.942388 4285 generic.go:356] "Generic (PLEG): container finished" podID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerID="366d657f25141c6683b99c57ae64e3fe0818ddb30225f79293670b4e76f15853" exitCode=0 Nov 02 22:31:53.942846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:53.942424 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" event={"ID":"bdc51def-3ffd-42be-9d56-2ca149cb4616","Type":"ContainerDied","Data":"366d657f25141c6683b99c57ae64e3fe0818ddb30225f79293670b4e76f15853"} Nov 02 22:31:54.008063 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.008040 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:31:54.065876 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.065845 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj\" (UniqueName: \"kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj\") pod \"bdc51def-3ffd-42be-9d56-2ca149cb4616\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " Nov 02 22:31:54.066018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.065991 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume\") pod \"bdc51def-3ffd-42be-9d56-2ca149cb4616\" (UID: \"bdc51def-3ffd-42be-9d56-2ca149cb4616\") " Nov 02 22:31:54.066410 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.066383 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume" pod "bdc51def-3ffd-42be-9d56-2ca149cb4616" (UID: "bdc51def-3ffd-42be-9d56-2ca149cb4616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:54.069102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.069076 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj" pod "bdc51def-3ffd-42be-9d56-2ca149cb4616" (UID: "bdc51def-3ffd-42be-9d56-2ca149cb4616"). InnerVolumeSpecName "kube-api-access-m4zbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:54.094936 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.094916 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:31:54.167476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.167455 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb\" (UniqueName: \"kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb\") pod \"879f512b-285e-4719-a70d-9fc95a414d1a\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " Nov 02 22:31:54.167616 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.167560 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume\") pod \"879f512b-285e-4719-a70d-9fc95a414d1a\" (UID: \"879f512b-285e-4719-a70d-9fc95a414d1a\") " Nov 02 22:31:54.167874 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.167848 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc51def-3ffd-42be-9d56-2ca149cb4616-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:54.167874 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.167864 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume" pod "879f512b-285e-4719-a70d-9fc95a414d1a" (UID: "879f512b-285e-4719-a70d-9fc95a414d1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:54.167874 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.167874 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zbj\" (UniqueName: \"kubernetes.io/projected/bdc51def-3ffd-42be-9d56-2ca149cb4616-kube-api-access-m4zbj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:54.169492 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.169474 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb" pod "879f512b-285e-4719-a70d-9fc95a414d1a" (UID: "879f512b-285e-4719-a70d-9fc95a414d1a"). InnerVolumeSpecName "kube-api-access-dbpsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:54.268867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.268844 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpsb\" (UniqueName: \"kubernetes.io/projected/879f512b-285e-4719-a70d-9fc95a414d1a-kube-api-access-dbpsb\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:54.268867 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.268866 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879f512b-285e-4719-a70d-9fc95a414d1a-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:31:54.948272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.948239 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" Nov 02 22:31:54.948742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.948247 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l" event={"ID":"bdc51def-3ffd-42be-9d56-2ca149cb4616","Type":"ContainerDied","Data":"d0fead7dae7a37bc18c38f3096113e33385ac1ddc7bcff60b5dba1f58fff2d6c"} Nov 02 22:31:54.948742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.948372 4285 scope.go:122] "RemoveContainer" containerID="366d657f25141c6683b99c57ae64e3fe0818ddb30225f79293670b4e76f15853" Nov 02 22:31:54.949741 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.949720 4285 generic.go:356] "Generic (PLEG): container finished" podID="879f512b-285e-4719-a70d-9fc95a414d1a" containerID="f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2" exitCode=0 Nov 02 22:31:54.949847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.949829 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" Nov 02 22:31:54.949937 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.949830 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" event={"ID":"879f512b-285e-4719-a70d-9fc95a414d1a","Type":"ContainerDied","Data":"f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2"} Nov 02 22:31:54.949988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.949935 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv" event={"ID":"879f512b-285e-4719-a70d-9fc95a414d1a","Type":"ContainerDied","Data":"32cb0867dca8b1e2d8e0ec2d1a7eba98fbc8aba52872ad2c0d16dd9801a5b8bd"} Nov 02 22:31:54.958250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.958234 4285 scope.go:122] "RemoveContainer" containerID="f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2" Nov 02 22:31:54.965908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.965894 4285 scope.go:122] "RemoveContainer" containerID="f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2" Nov 02 22:31:54.966183 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:31:54.966156 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2\": container with ID starting with f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2 not found: ID does not exist" containerID="f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2" Nov 02 22:31:54.966248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:54.966182 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2"} err="failed to get container status \"f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2\": rpc error: code = NotFound desc = could not find container \"f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2\": container with ID starting with f69d4e2cf7d58f66c8e1816c382f75d04cac7ef3d025d802b821a90969473ad2 not found: ID does not exist" Nov 02 22:31:55.110288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:55.110229 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l"] Nov 02 22:31:55.113486 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:55.113455 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-53bfd6-5cc4f6bfffvr8l"] Nov 02 22:31:55.171337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:55.171311 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" path="/var/lib/kubelet/pods/bdc51def-3ffd-42be-9d56-2ca149cb4616/volumes" Nov 02 22:31:55.221511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:55.221488 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv"] Nov 02 22:31:55.225274 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:55.225254 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-53bfd6-f74ccdbcvv"] Nov 02 22:31:57.171697 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:57.171664 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" path="/var/lib/kubelet/pods/879f512b-285e-4719-a70d-9fc95a414d1a/volumes" Nov 02 22:31:58.128337 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:58.128307 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:59.204851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:31:59.204818 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:02.320847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.320812 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk"] Nov 02 22:32:02.321271 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.321027 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" containerID="cri-o://c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2" gracePeriod=10 Nov 02 22:32:02.437809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.437770 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused" Nov 02 22:32:02.571377 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.571323 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 22:32:02.644383 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.644355 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume\") pod \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " Nov 02 22:32:02.644518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.644396 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq\" (UniqueName: \"kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq\") pod \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\" (UID: \"1b5549b0-1277-4bbe-9600-2a9eb00575a0\") " Nov 02 22:32:02.644769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.644743 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume" pod "1b5549b0-1277-4bbe-9600-2a9eb00575a0" (UID: "1b5549b0-1277-4bbe-9600-2a9eb00575a0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:02.644883 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.644840 4285 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b5549b0-1277-4bbe-9600-2a9eb00575a0-extensions-socket-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.646414 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.646391 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq" pod "1b5549b0-1277-4bbe-9600-2a9eb00575a0" (UID: "1b5549b0-1277-4bbe-9600-2a9eb00575a0"). InnerVolumeSpecName "kube-api-access-mr6dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:02.745635 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.745594 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6dq\" (UniqueName: \"kubernetes.io/projected/1b5549b0-1277-4bbe-9600-2a9eb00575a0-kube-api-access-mr6dq\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.995289 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.995256 4285 generic.go:356] "Generic (PLEG): container finished" podID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerID="c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2" exitCode=0 Nov 02 22:32:02.995459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.995315 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" Nov 02 22:32:02.995459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.995339 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" event={"ID":"1b5549b0-1277-4bbe-9600-2a9eb00575a0","Type":"ContainerDied","Data":"c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2"} Nov 02 22:32:02.995459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.995399 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk" event={"ID":"1b5549b0-1277-4bbe-9600-2a9eb00575a0","Type":"ContainerDied","Data":"445c209fa8b456fdd46d7766aa668b4cff8ed2ceba0a2d6feacd31f263e48dd7"} Nov 02 22:32:02.995459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:02.995428 4285 scope.go:122] "RemoveContainer" containerID="c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2" Nov 02 22:32:03.007698 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.007677 4285 scope.go:122] "RemoveContainer" containerID="7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4" Nov 02 22:32:03.015426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.015410 4285 scope.go:122] "RemoveContainer" containerID="c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2" Nov 02 22:32:03.015676 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:32:03.015658 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2\": container with ID starting with c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2 not found: ID does not exist" containerID="c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2" Nov 02 22:32:03.015716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.015685 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2"} err="failed to get container status \"c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2\": rpc error: code = NotFound desc = could not find container \"c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2\": container with ID starting with c5f3e72c3e46b8e1e3779bdd6c469ab194393ce49bd5149b15dff92e6698d5f2 not found: ID does not exist" Nov 02 22:32:03.015716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.015703 4285 scope.go:122] "RemoveContainer" containerID="7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4" Nov 02 22:32:03.015925 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:32:03.015906 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4\": container with ID starting with 7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4 not found: ID does not exist" containerID="7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4" Nov 02 22:32:03.015965 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.015931 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4"} err="failed to get container status \"7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4\": rpc error: code = NotFound desc = could not find container \"7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4\": container with ID starting with 7d70a01630fc21c688646e45210bf6f7833338d2e0434e292aa48428f42d3cc4 not found: ID does not exist" Nov 02 22:32:03.159026 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.159001 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk"] Nov 02 22:32:03.208358 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:03.208331 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sq5jk"] Nov 02 22:32:04.835339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:04.835256 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:05.171385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:05.171313 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" path="/var/lib/kubelet/pods/1b5549b0-1277-4bbe-9600-2a9eb00575a0/volumes" Nov 02 22:32:06.044490 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:06.044456 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r"] Nov 02 22:32:06.093785 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:06.093757 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nk76r"] Nov 02 22:32:07.171905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:07.171872 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e7c846-c3ad-455a-a4d3-2330800637ee" path="/var/lib/kubelet/pods/41e7c846-c3ad-455a-a4d3-2330800637ee/volumes" Nov 02 22:32:16.004050 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:32:16.004015 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:32:21.755386 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:21.755341 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:24.587355 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:24.587325 4285 scope.go:122] "RemoveContainer" containerID="65498b90e12facd8f1824e1db1e4402939168a30348e1af3fcfa6e2510c4ebc1" Nov 02 22:32:24.595417 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:24.595398 4285 scope.go:122] "RemoveContainer" containerID="0e22638d521026c24fdbf1375cf7acd8215b0e758e0e2183e312e273d3e6054f" Nov 02 22:32:37.043898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:37.043862 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp"] Nov 02 22:32:37.093146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:37.093107 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6zpzp"] Nov 02 22:32:37.171665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:32:37.171633 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63415894-08ee-40c8-94ac-05191ce39c04" path="/var/lib/kubelet/pods/63415894-08ee-40c8-94ac-05191ce39c04/volumes" Nov 02 22:32:46.004136 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:32:46.004085 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:33:00.045702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:00.045663 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl"] Nov 02 22:33:00.094743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:00.094716 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-z9cnl"] Nov 02 22:33:01.171703 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:01.171672 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4135bc6-a5c1-4168-b50c-44f90ea3b6c5" path="/var/lib/kubelet/pods/d4135bc6-a5c1-4168-b50c-44f90ea3b6c5/volumes" Nov 02 22:33:05.962501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962470 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm"] Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962829 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962839 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962849 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962853 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962864 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962869 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962876 4285 cpu_manager.go:414] "containerMap: removing container" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerName="payload-processing" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962880 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerName="payload-processing" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962893 4285 cpu_manager.go:414] "containerMap: removing container" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" containerName="payload-pre-processing" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962897 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" containerName="payload-pre-processing" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962904 4285 cpu_manager.go:414] "containerMap: removing container" podUID="4c8fdaae-67a2-47b9-9581-061212594a97" containerName="revoke-keys" Nov 02 22:33:05.962958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962908 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8fdaae-67a2-47b9-9581-061212594a97" containerName="revoke-keys" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962970 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8fdaae-67a2-47b9-9581-061212594a97" containerName="revoke-keys" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962981 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962988 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.962994 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.963000 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5549b0-1277-4bbe-9600-2a9eb00575a0" containerName="manager" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.963005 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc51def-3ffd-42be-9d56-2ca149cb4616" containerName="payload-processing" Nov 02 22:33:05.963366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.963011 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="879f512b-285e-4719-a70d-9fc95a414d1a" containerName="payload-pre-processing" Nov 02 22:33:05.966244 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:05.966226 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.018612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.018558 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:33:06.018612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.018599 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:33:06.018896 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.018880 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lcwpm\"" Nov 02 22:33:06.070557 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.070531 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm"] Nov 02 22:33:06.084194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.084173 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpk92\" (UniqueName: \"kubernetes.io/projected/3a657361-0cf8-460f-9f1c-2cb6dc0cc194-kube-api-access-gpk92\") pod \"kuadrant-operator-controller-manager-678d87766f-kdxrm\" (UID: \"3a657361-0cf8-460f-9f1c-2cb6dc0cc194\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.084309 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.084216 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a657361-0cf8-460f-9f1c-2cb6dc0cc194-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kdxrm\" (UID: \"3a657361-0cf8-460f-9f1c-2cb6dc0cc194\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.185221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.185180 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpk92\" (UniqueName: \"kubernetes.io/projected/3a657361-0cf8-460f-9f1c-2cb6dc0cc194-kube-api-access-gpk92\") pod \"kuadrant-operator-controller-manager-678d87766f-kdxrm\" (UID: \"3a657361-0cf8-460f-9f1c-2cb6dc0cc194\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.185409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.185234 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a657361-0cf8-460f-9f1c-2cb6dc0cc194-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kdxrm\" (UID: \"3a657361-0cf8-460f-9f1c-2cb6dc0cc194\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.185578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.185560 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a657361-0cf8-460f-9f1c-2cb6dc0cc194-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kdxrm\" (UID: \"3a657361-0cf8-460f-9f1c-2cb6dc0cc194\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.246227 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.246158 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpk92\" (UniqueName: \"kubernetes.io/projected/3a657361-0cf8-460f-9f1c-2cb6dc0cc194-kube-api-access-gpk92\") pod \"kuadrant-operator-controller-manager-678d87766f-kdxrm\" (UID: \"3a657361-0cf8-460f-9f1c-2cb6dc0cc194\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.278587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.278562 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:06.699395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:06.699371 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm"] Nov 02 22:33:06.701796 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:33:06.701769 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a657361_0cf8_460f_9f1c_2cb6dc0cc194.slice/crio-0a48eddb8950d16c7ec15ede21909966c312a9a70905d5a3f524c15a3c77d6f3 WatchSource:0}: Error finding container 0a48eddb8950d16c7ec15ede21909966c312a9a70905d5a3f524c15a3c77d6f3: Status 404 returned error can't find the container with id 0a48eddb8950d16c7ec15ede21909966c312a9a70905d5a3f524c15a3c77d6f3 Nov 02 22:33:07.299644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:07.299602 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" event={"ID":"3a657361-0cf8-460f-9f1c-2cb6dc0cc194","Type":"ContainerStarted","Data":"7091442b6e5f14d59df628eb492d43a992aec78c4978ce1f5c179720695deef0"} Nov 02 22:33:07.300037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:07.299666 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" event={"ID":"3a657361-0cf8-460f-9f1c-2cb6dc0cc194","Type":"ContainerStarted","Data":"0a48eddb8950d16c7ec15ede21909966c312a9a70905d5a3f524c15a3c77d6f3"} Nov 02 22:33:07.300037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:07.299694 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:07.415944 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:07.415895 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" podStartSLOduration=2.415883267 podStartE2EDuration="2.415883267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:33:07.413602725 +0000 UTC m=+3526.375153041" watchObservedRunningTime="2026-11-02 22:33:07.415883267 +0000 UTC m=+3526.377433585" Nov 02 22:33:16.005304 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:33:16.005268 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:33:18.307271 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:18.307238 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kdxrm" Nov 02 22:33:22.095240 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:22.095200 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:24.634802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:24.634773 4285 scope.go:122] "RemoveContainer" containerID="383266578cb52e42193376bab4f6d22cb04ea86e5ff7f5844a477daabbc3fa32" Nov 02 22:33:24.643061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:24.643038 4285 scope.go:122] "RemoveContainer" containerID="efa4ee025b72fa3615298edd0219cff8cd34726f4577cd4752a23e9fa3df8d3a" Nov 02 22:33:29.469265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:29.469226 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-mcx5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:33:29.469870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:29.469838 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-mcx5l" podUID="cd208f3f-912d-4600-982d-22c0ff27389e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:33:29.490245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:29.490213 4285 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-blcrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:33:29.490482 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:29.490453 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-blcrl" podUID="6eb34991-dfd5-4e0f-9639-0aad1f6f4447" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:33:31.613732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.613699 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f"] Nov 02 22:33:31.614155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.614076 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" Nov 02 22:33:31.614155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.614087 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5dfeb4-858b-4e30-a69d-cec9d22f12ed" containerName="istio-proxy" Nov 02 22:33:31.617150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.617113 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.669228 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.669201 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:33:31.718905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.718875 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:33:31.721048 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.721029 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f"] Nov 02 22:33:31.725291 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.725274 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mdt\" (UniqueName: \"kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.725344 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.725312 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.725493 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.725479 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.826972 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.826940 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mdt\" (UniqueName: \"kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.827167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.826984 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.827167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.827054 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.827818 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.827797 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.829390 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.829370 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.883273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.883219 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mdt\" (UniqueName: \"kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt\") pod \"maas-api-revoke-sub-d576e68b-ft62f\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:31.929235 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:31.929213 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:32.299587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:32.299545 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f"] Nov 02 22:33:32.306468 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:33:32.306433 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeeb90a_41a2_48f4_ac46_703dc3ebe4f1.slice/crio-f44b0549b3168a7a2e95791b912142e3d084cdaa7a759e9a92ea949f8a85eac7 WatchSource:0}: Error finding container f44b0549b3168a7a2e95791b912142e3d084cdaa7a759e9a92ea949f8a85eac7: Status 404 returned error can't find the container with id f44b0549b3168a7a2e95791b912142e3d084cdaa7a759e9a92ea949f8a85eac7 Nov 02 22:33:32.441287 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:32.441252 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" event={"ID":"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1","Type":"ContainerStarted","Data":"f44b0549b3168a7a2e95791b912142e3d084cdaa7a759e9a92ea949f8a85eac7"} Nov 02 22:33:33.448678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:33.448636 4285 generic.go:356] "Generic (PLEG): container finished" podID="fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" containerID="d98f31bfea73ec85d7153f6b363e57c16053de8a2a630f6d6c61de53e49e3c27" exitCode=0 Nov 02 22:33:33.448678 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:33.448664 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" event={"ID":"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1","Type":"ContainerDied","Data":"d98f31bfea73ec85d7153f6b363e57c16053de8a2a630f6d6c61de53e49e3c27"} Nov 02 22:33:34.579845 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.579821 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:34.655398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.655373 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt\" (UniqueName: \"kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt\") pod \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " Nov 02 22:33:34.655507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.655462 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca\") pod \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " Nov 02 22:33:34.655558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.655513 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca\") pod \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\" (UID: \"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1\") " Nov 02 22:33:34.655813 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.655787 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca" pod "fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" (UID: "fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.657426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.657400 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca" pod "fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" (UID: "fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:34.657524 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.657479 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt" pod "fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" (UID: "fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1"). InnerVolumeSpecName "kube-api-access-47mdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.756693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.756630 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mdt\" (UniqueName: \"kubernetes.io/projected/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-kube-api-access-47mdt\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.756693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.756654 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.756693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:34.756665 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.459448 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:35.459416 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" event={"ID":"fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1","Type":"ContainerDied","Data":"f44b0549b3168a7a2e95791b912142e3d084cdaa7a759e9a92ea949f8a85eac7"} Nov 02 22:33:35.459610 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:35.459448 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f" Nov 02 22:33:35.459610 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:33:35.459467 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f44b0549b3168a7a2e95791b912142e3d084cdaa7a759e9a92ea949f8a85eac7" Nov 02 22:33:46.005238 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:33:46.005205 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:34:00.598234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.598196 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh"] Nov 02 22:34:00.598806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.598786 4285 cpu_manager.go:414] "containerMap: removing container" podUID="fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" containerName="revoke-keys" Nov 02 22:34:00.598892 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.598808 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" containerName="revoke-keys" Nov 02 22:34:00.598949 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.598896 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" containerName="revoke-keys" Nov 02 22:34:00.601907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.601887 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.656652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.656621 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:34:00.700339 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.700303 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.700518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.700353 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.700518 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.700374 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkpt\" (UniqueName: \"kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.706191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.706169 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:34:00.708199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.708179 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh"] Nov 02 22:34:00.801216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.801180 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.801365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.801236 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.801365 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.801262 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkpt\" (UniqueName: \"kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.801941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.801916 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.803474 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.803453 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.857474 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.857414 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkpt\" (UniqueName: \"kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt\") pod \"maas-api-revoke-sub-d2bc93bc-vkjsh\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:00.914742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:00.914718 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:01.286969 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:01.286942 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh"] Nov 02 22:34:01.289208 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:34:01.289178 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cd755d_eb95_469c_a709_681d3c32642f.slice/crio-9d7412e1178fa1fc26c191c5c62cde11e9e7f0ad06320471a897108e51277ab1 WatchSource:0}: Error finding container 9d7412e1178fa1fc26c191c5c62cde11e9e7f0ad06320471a897108e51277ab1: Status 404 returned error can't find the container with id 9d7412e1178fa1fc26c191c5c62cde11e9e7f0ad06320471a897108e51277ab1 Nov 02 22:34:01.623146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:01.623092 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" event={"ID":"44cd755d-eb95-469c-a709-681d3c32642f","Type":"ContainerStarted","Data":"16cc6783eb15da27f499982c029e8ce369d743ec3646d464d319adb0c3e14a13"} Nov 02 22:34:01.623566 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:01.623205 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" event={"ID":"44cd755d-eb95-469c-a709-681d3c32642f","Type":"ContainerStarted","Data":"9d7412e1178fa1fc26c191c5c62cde11e9e7f0ad06320471a897108e51277ab1"} Nov 02 22:34:01.743028 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:01.742974 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" podStartSLOduration=1.742960626 podStartE2EDuration="1.742960626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:01.740398437 +0000 UTC m=+3580.701948756" watchObservedRunningTime="2026-11-02 22:34:01.742960626 +0000 UTC m=+3580.704510945" Nov 02 22:34:02.629771 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:02.629687 4285 generic.go:356] "Generic (PLEG): container finished" podID="44cd755d-eb95-469c-a709-681d3c32642f" containerID="16cc6783eb15da27f499982c029e8ce369d743ec3646d464d319adb0c3e14a13" exitCode=0 Nov 02 22:34:02.630147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:02.629770 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" event={"ID":"44cd755d-eb95-469c-a709-681d3c32642f","Type":"ContainerDied","Data":"16cc6783eb15da27f499982c029e8ce369d743ec3646d464d319adb0c3e14a13"} Nov 02 22:34:03.760372 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.760346 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:03.833042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.833012 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt\" (UniqueName: \"kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt\") pod \"44cd755d-eb95-469c-a709-681d3c32642f\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " Nov 02 22:34:03.833220 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.833138 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca\") pod \"44cd755d-eb95-469c-a709-681d3c32642f\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " Nov 02 22:34:03.833272 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.833229 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca\") pod \"44cd755d-eb95-469c-a709-681d3c32642f\" (UID: \"44cd755d-eb95-469c-a709-681d3c32642f\") " Nov 02 22:34:03.833475 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.833450 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca" pod "44cd755d-eb95-469c-a709-681d3c32642f" (UID: "44cd755d-eb95-469c-a709-681d3c32642f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:03.834965 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.834948 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca" pod "44cd755d-eb95-469c-a709-681d3c32642f" (UID: "44cd755d-eb95-469c-a709-681d3c32642f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:03.835080 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.835064 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt" pod "44cd755d-eb95-469c-a709-681d3c32642f" (UID: "44cd755d-eb95-469c-a709-681d3c32642f"). InnerVolumeSpecName "kube-api-access-fmkpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:03.934116 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.934059 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:34:03.934116 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.934081 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkpt\" (UniqueName: \"kubernetes.io/projected/44cd755d-eb95-469c-a709-681d3c32642f-kube-api-access-fmkpt\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:34:03.934116 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:03.934094 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cd755d-eb95-469c-a709-681d3c32642f-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:34:04.640742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:04.640707 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" event={"ID":"44cd755d-eb95-469c-a709-681d3c32642f","Type":"ContainerDied","Data":"9d7412e1178fa1fc26c191c5c62cde11e9e7f0ad06320471a897108e51277ab1"} Nov 02 22:34:04.640742 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:04.640733 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh" Nov 02 22:34:04.640996 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:04.640752 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7412e1178fa1fc26c191c5c62cde11e9e7f0ad06320471a897108e51277ab1" Nov 02 22:34:08.046644 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:08.046611 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx"] Nov 02 22:34:08.095763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:08.095733 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vhbpx"] Nov 02 22:34:09.171426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:09.171393 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae5f21d-b0e1-4bc3-a0f0-87a599f32736" path="/var/lib/kubelet/pods/9ae5f21d-b0e1-4bc3-a0f0-87a599f32736/volumes" Nov 02 22:34:09.844230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:09.844185 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:13.010303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.010263 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:13.484748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.484710 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc"] Nov 02 22:34:13.485208 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.485187 4285 cpu_manager.go:414] "containerMap: removing container" podUID="44cd755d-eb95-469c-a709-681d3c32642f" containerName="revoke-keys" Nov 02 22:34:13.485280 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.485209 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cd755d-eb95-469c-a709-681d3c32642f" containerName="revoke-keys" Nov 02 22:34:13.485322 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.485286 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cd755d-eb95-469c-a709-681d3c32642f" containerName="revoke-keys" Nov 02 22:34:13.491774 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.490263 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.554305 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.554271 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8b2f0b-openshift-default-dockercfg-dd7lx\"" Nov 02 22:34:13.602415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.602381 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc"] Nov 02 22:34:13.624690 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.624654 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ndq\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.624852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.624708 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.624852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.624777 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.624852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.624830 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.625037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.624920 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.625037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.624976 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.625037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.625014 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.625206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.625053 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.625206 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.625178 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.719954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.719922 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:13.726134 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726101 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726224 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726151 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726224 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726187 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726224 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726215 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726255 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726276 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726302 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726347 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726470 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726528 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ndq\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726536 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726648 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.726775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.726698 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.727053 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.727029 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.728629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.728607 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.728819 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.728799 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.788423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.788346 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.788423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.788357 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ndq\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq\") pod \"e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:13.804248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:13.804221 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:14.006909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.006875 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:14.230493 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.230459 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc"] Nov 02 22:34:14.268022 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:34:14.267996 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a26cc0_6354_4fdd_8990_be2e270c4e4f.slice/crio-683229ef886ec23d26b5927984426a74b112e543d11f078435cee9acb69f0f60 WatchSource:0}: Error finding container 683229ef886ec23d26b5927984426a74b112e543d11f078435cee9acb69f0f60: Status 404 returned error can't find the container with id 683229ef886ec23d26b5927984426a74b112e543d11f078435cee9acb69f0f60 Nov 02 22:34:14.269953 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.269926 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:14.270020 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.269989 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:14.270020 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.270015 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:14.691802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.691725 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" event={"ID":"a7a26cc0-6354-4fdd-8990-be2e270c4e4f","Type":"ContainerStarted","Data":"536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755"} Nov 02 22:34:14.691802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.691776 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" event={"ID":"a7a26cc0-6354-4fdd-8990-be2e270c4e4f","Type":"ContainerStarted","Data":"683229ef886ec23d26b5927984426a74b112e543d11f078435cee9acb69f0f60"} Nov 02 22:34:14.804670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.804645 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:14.809989 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:14.809947 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" podStartSLOduration=1.8099343380000001 podStartE2EDuration="1.809934338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:14.807924119 +0000 UTC m=+3593.769474455" watchObservedRunningTime="2026-11-02 22:34:14.809934338 +0000 UTC m=+3593.771484658" Nov 02 22:34:15.809744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:15.809719 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:16.006966 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:34:16.006933 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:34:16.702253 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:16.702218 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:16.702925 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:16.702906 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:17.681239 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:17.681209 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:17.857444 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:17.857412 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:18.054551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:18.054523 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp"] Nov 02 22:34:18.103708 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:18.103683 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-rfvsp"] Nov 02 22:34:19.176631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:19.176597 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab7cbd2-3a71-4876-81ff-7a4eda94c851" path="/var/lib/kubelet/pods/4ab7cbd2-3a71-4876-81ff-7a4eda94c851/volumes" Nov 02 22:34:20.894435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:20.894375 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:21.063507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.063476 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:21.274473 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.274441 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9"] Nov 02 22:34:21.278375 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.278360 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.384304 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.384265 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-79d38df1-dockercfg-jl7rj\"" Nov 02 22:34:21.384304 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.384291 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8b2f0b\"" Nov 02 22:34:21.385790 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.385766 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9"] Nov 02 22:34:21.398302 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.398279 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4578m\" (UniqueName: \"kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m\") pod \"payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.398431 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.398321 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.434200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.434177 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb"] Nov 02 22:34:21.437704 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.437688 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:21.489489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.489466 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb"] Nov 02 22:34:21.499908 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.499882 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4578m\" (UniqueName: \"kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m\") pod \"payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.500045 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.499954 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.551647 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.551563 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8b2f0b\"" Nov 02 22:34:21.556852 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.556832 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4578m\" (UniqueName: \"kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m\") pod \"payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.561040 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.561017 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.601187 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.601162 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:21.601288 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.601236 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4rz\" (UniqueName: \"kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz\") pod \"payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:21.641640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.641617 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-79d38df1-dockercfg-jl7rj\"" Nov 02 22:34:21.651702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.651675 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:21.702254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.702215 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4rz\" (UniqueName: \"kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz\") pod \"payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:21.702427 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.702362 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:21.703312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.703221 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:21.758720 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:21.758637 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4rz\" (UniqueName: \"kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz\") pod \"payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:22.024578 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.024553 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9"] Nov 02 22:34:22.026160 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:34:22.026118 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8f2f90_e807_4afe_86d5_559279ee81be.slice/crio-38894f1240a314c63903e00b619fa2df5176452a1d2a4f793b65c6230a029b91 WatchSource:0}: Error finding container 38894f1240a314c63903e00b619fa2df5176452a1d2a4f793b65c6230a029b91: Status 404 returned error can't find the container with id 38894f1240a314c63903e00b619fa2df5176452a1d2a4f793b65c6230a029b91 Nov 02 22:34:22.052268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.052248 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:22.420478 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.420452 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb"] Nov 02 22:34:22.421735 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.421706 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:22.422209 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:34:22.422186 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f8fa7_4bf8_4915_b01f_b18f9fcd60c3.slice/crio-81edd9f3de4cc20a60f4cba31a96c433d1f334de0ac073109422ac0056625157 WatchSource:0}: Error finding container 81edd9f3de4cc20a60f4cba31a96c433d1f334de0ac073109422ac0056625157: Status 404 returned error can't find the container with id 81edd9f3de4cc20a60f4cba31a96c433d1f334de0ac073109422ac0056625157 Nov 02 22:34:22.734117 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.734076 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" event={"ID":"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3","Type":"ContainerStarted","Data":"c8c6534b1b24d3ca96d0d5adc70c719a6a99c345824967ec21ce14c5b7f94109"} Nov 02 22:34:22.734301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.734160 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:22.734301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.734191 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" event={"ID":"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3","Type":"ContainerStarted","Data":"81edd9f3de4cc20a60f4cba31a96c433d1f334de0ac073109422ac0056625157"} Nov 02 22:34:22.735553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.735531 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" event={"ID":"0b8f2f90-e807-4afe-86d5-559279ee81be","Type":"ContainerStarted","Data":"74627acbbf0e592ae8ada5c1b0cbeacbbccda10718b9245000be10e63b14472b"} Nov 02 22:34:22.735671 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.735568 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" event={"ID":"0b8f2f90-e807-4afe-86d5-559279ee81be","Type":"ContainerStarted","Data":"38894f1240a314c63903e00b619fa2df5176452a1d2a4f793b65c6230a029b91"} Nov 02 22:34:22.735671 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.735587 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:22.847924 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.847870 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" podStartSLOduration=1.8478553340000001 podStartE2EDuration="1.847855334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:22.847750407 +0000 UTC m=+3601.809300726" watchObservedRunningTime="2026-11-02 22:34:22.847855334 +0000 UTC m=+3601.809405653" Nov 02 22:34:22.970366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:22.969474 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" podStartSLOduration=1.9694559740000002 podStartE2EDuration="1.969455974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:22.969223095 +0000 UTC m=+3601.930773411" watchObservedRunningTime="2026-11-02 22:34:22.969455974 +0000 UTC m=+3601.931006295" Nov 02 22:34:23.954137 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:23.954102 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:24.665037 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:24.665007 4285 scope.go:122] "RemoveContainer" containerID="a0fc1cb6e75f9cdef76097988cdf06feb7e85ee5ca3097bf7725ede636a23370" Nov 02 22:34:24.674545 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:24.674526 4285 scope.go:122] "RemoveContainer" containerID="38082e780bab2472a90979798f57abfafeee148803692991432a57753d1a9cce" Nov 02 22:34:24.684082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:24.684053 4285 scope.go:122] "RemoveContainer" containerID="00ca50f94c85321db7d14dcd522339ca8e6e74dc320ce80c23aae5ce8f868357" Nov 02 22:34:25.889086 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:25.889047 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:28.913353 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:28.913321 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:33.725110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:33.725023 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:33.746837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:33.746809 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:34:33.751423 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:33.751363 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:34:38.507251 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:38.507214 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:40.864326 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:40.864298 4285 generic.go:356] "Generic (PLEG): container finished" podID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerID="536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755" exitCode=0 Nov 02 22:34:40.864738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:40.864371 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" event={"ID":"a7a26cc0-6354-4fdd-8990-be2e270c4e4f","Type":"ContainerDied","Data":"536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755"} Nov 02 22:34:40.864738 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:40.864705 4285 scope.go:122] "RemoveContainer" containerID="536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755" Nov 02 22:34:40.865777 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:40.865748 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:40.865851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:40.865818 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:40.865905 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:40.865853 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:41.871223 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:41.871186 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" event={"ID":"a7a26cc0-6354-4fdd-8990-be2e270c4e4f","Type":"ContainerStarted","Data":"698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d"} Nov 02 22:34:42.482543 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:42.482513 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:42.804772 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:42.804700 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:42.805502 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:42.805484 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 22:34:42.805560 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:42.805529 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 22:34:42.976982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:42.976948 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:43.153411 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:43.153328 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:43.737761 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:43.737684 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:43.804553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:43.804525 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 22:34:43.804731 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:43.804573 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 22:34:44.805015 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:44.804983 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 22:34:44.805572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:44.805034 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 22:34:45.805144 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:45.805092 4285 patch_prober.go:29] interesting pod/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 02 22:34:45.805551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:45.805158 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 02 22:34:46.004614 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:34:46.004578 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:34:46.275866 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:46.275835 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:46.435565 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:46.435531 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:46.706348 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:46.706314 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:46.808415 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:46.808389 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:46.896564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:46.896537 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:34:48.417209 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:48.417162 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:53.226759 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:53.226724 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:58.008432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:34:58.008398 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:02.851469 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:02.847305 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:07.728978 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:07.728898 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:12.526102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:12.526069 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:16.012861 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:35:16.012794 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:35:17.324047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:17.323998 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:22.164055 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:22.164023 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:22.754680 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:22.754637 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:24.709999 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:24.709969 4285 scope.go:122] "RemoveContainer" containerID="0329aee5f1e70f7a27372cca7ce8d3f00fdb0a656b09f38580a30e18f02f8e64" Nov 02 22:35:24.718441 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:24.718423 4285 scope.go:122] "RemoveContainer" containerID="bb3d5b53c11afa54ae162aeae8ba6bf09544c212f78ae3bdde20e4884f15acea" Nov 02 22:35:24.725489 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:24.725467 4285 scope.go:122] "RemoveContainer" containerID="787e38e2a6dcf02968de13ed956cbf3c978c6a6ae36a6248ebe313aa94b63f99" Nov 02 22:35:26.835104 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:26.835066 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:31.059426 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.059395 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7"] Nov 02 22:35:31.062950 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.062933 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.117314 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.117289 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:35:31.120149 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.120097 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7"] Nov 02 22:35:31.166732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.166707 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:35:31.202757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.202728 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2rj\" (UniqueName: \"kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.202889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.202867 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.202932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.202901 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.303943 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.303914 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.304079 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.303953 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.304079 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.303991 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2rj\" (UniqueName: \"kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.304794 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.304766 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.306250 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.306229 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.360757 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.360702 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2rj\" (UniqueName: \"kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj\") pod \"maas-api-revoke-sub-2cf2a4ca-g45v7\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.375547 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.375528 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:31.637512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.637436 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:31.746354 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:31.746329 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7"] Nov 02 22:35:31.748189 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:35:31.748163 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9caa702_5ca2_4614_88d9_1506d6858ea4.slice/crio-b328e8c828a8e706b58ec2612adedd5384541997cd98a3a5674996b6c744f83b WatchSource:0}: Error finding container b328e8c828a8e706b58ec2612adedd5384541997cd98a3a5674996b6c744f83b: Status 404 returned error can't find the container with id b328e8c828a8e706b58ec2612adedd5384541997cd98a3a5674996b6c744f83b Nov 02 22:35:32.127981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:32.127949 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" event={"ID":"d9caa702-5ca2-4614-88d9-1506d6858ea4","Type":"ContainerStarted","Data":"c6a45c9c105662571a441df0bc2f8711acd6738be0885a6d21f768c68401b9ad"} Nov 02 22:35:32.128385 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:32.128002 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" event={"ID":"d9caa702-5ca2-4614-88d9-1506d6858ea4","Type":"ContainerStarted","Data":"b328e8c828a8e706b58ec2612adedd5384541997cd98a3a5674996b6c744f83b"} Nov 02 22:35:32.242155 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:32.242075 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" podStartSLOduration=1.242060932 podStartE2EDuration="1.242060932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:32.240394785 +0000 UTC m=+3671.201945121" watchObservedRunningTime="2026-11-02 22:35:32.242060932 +0000 UTC m=+3671.203611284" Nov 02 22:35:33.134697 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:33.134660 4285 generic.go:356] "Generic (PLEG): container finished" podID="d9caa702-5ca2-4614-88d9-1506d6858ea4" containerID="c6a45c9c105662571a441df0bc2f8711acd6738be0885a6d21f768c68401b9ad" exitCode=0 Nov 02 22:35:33.135098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:33.134742 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" event={"ID":"d9caa702-5ca2-4614-88d9-1506d6858ea4","Type":"ContainerDied","Data":"c6a45c9c105662571a441df0bc2f8711acd6738be0885a6d21f768c68401b9ad"} Nov 02 22:35:34.277921 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.277897 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:34.435226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.435154 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca\") pod \"d9caa702-5ca2-4614-88d9-1506d6858ea4\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " Nov 02 22:35:34.435363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.435294 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca\") pod \"d9caa702-5ca2-4614-88d9-1506d6858ea4\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " Nov 02 22:35:34.435363 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.435332 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj\" (UniqueName: \"kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj\") pod \"d9caa702-5ca2-4614-88d9-1506d6858ea4\" (UID: \"d9caa702-5ca2-4614-88d9-1506d6858ea4\") " Nov 02 22:35:34.435625 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.435599 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca" pod "d9caa702-5ca2-4614-88d9-1506d6858ea4" (UID: "d9caa702-5ca2-4614-88d9-1506d6858ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:34.437315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.437289 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca" pod "d9caa702-5ca2-4614-88d9-1506d6858ea4" (UID: "d9caa702-5ca2-4614-88d9-1506d6858ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:34.437611 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.437583 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj" pod "d9caa702-5ca2-4614-88d9-1506d6858ea4" (UID: "d9caa702-5ca2-4614-88d9-1506d6858ea4"). InnerVolumeSpecName "kube-api-access-wl2rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:34.536897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.536872 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:35:34.536897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.536896 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl2rj\" (UniqueName: \"kubernetes.io/projected/d9caa702-5ca2-4614-88d9-1506d6858ea4-kube-api-access-wl2rj\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:35:34.537049 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:34.536905 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9caa702-5ca2-4614-88d9-1506d6858ea4-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:35:35.147816 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:35.147785 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" Nov 02 22:35:35.148018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:35.147819 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7" event={"ID":"d9caa702-5ca2-4614-88d9-1506d6858ea4","Type":"ContainerDied","Data":"b328e8c828a8e706b58ec2612adedd5384541997cd98a3a5674996b6c744f83b"} Nov 02 22:35:35.148018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:35.147891 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b328e8c828a8e706b58ec2612adedd5384541997cd98a3a5674996b6c744f83b" Nov 02 22:35:36.428203 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:36.428175 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:41.220683 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:41.220645 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:42.131861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.131829 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:42.293554 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.293522 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:42.668359 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.668331 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb"] Nov 02 22:35:42.668746 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.668734 4285 cpu_manager.go:414] "containerMap: removing container" podUID="d9caa702-5ca2-4614-88d9-1506d6858ea4" containerName="revoke-keys" Nov 02 22:35:42.668791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.668748 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9caa702-5ca2-4614-88d9-1506d6858ea4" containerName="revoke-keys" Nov 02 22:35:42.668831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.668810 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9caa702-5ca2-4614-88d9-1506d6858ea4" containerName="revoke-keys" Nov 02 22:35:42.671907 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.671891 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.773308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.773264 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:35:42.773459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.773441 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:35:42.775609 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.775587 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb"] Nov 02 22:35:42.816539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.816516 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkkq\" (UniqueName: \"kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.816693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.816602 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.816758 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.816717 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.918138 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.918099 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkkq\" (UniqueName: \"kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.918267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.918214 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.918267 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.918256 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.918896 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.918854 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.920402 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.920385 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.975010 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.974979 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkkq\" (UniqueName: \"kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:42.984861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:42.984836 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:43.354833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:43.354806 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb"] Nov 02 22:35:43.357508 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:35:43.357471 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode223aabb_8121_452e_970e_9ab379b5d300.slice/crio-73aebcb70814c2e31e3092bbc9ada3073b8bcd075af9d00b9a7ec4cfa1565450 WatchSource:0}: Error finding container 73aebcb70814c2e31e3092bbc9ada3073b8bcd075af9d00b9a7ec4cfa1565450: Status 404 returned error can't find the container with id 73aebcb70814c2e31e3092bbc9ada3073b8bcd075af9d00b9a7ec4cfa1565450 Nov 02 22:35:44.210615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:44.210575 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" event={"ID":"e223aabb-8121-452e-970e-9ab379b5d300","Type":"ContainerStarted","Data":"b42cb0c2be56631b1eb106b374d23a94b1a12aa1bc2649dd0d53c7cd42ce57a4"} Nov 02 22:35:44.210818 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:44.210641 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" event={"ID":"e223aabb-8121-452e-970e-9ab379b5d300","Type":"ContainerStarted","Data":"73aebcb70814c2e31e3092bbc9ada3073b8bcd075af9d00b9a7ec4cfa1565450"} Nov 02 22:35:44.325002 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:44.324951 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" podStartSLOduration=2.324937248 podStartE2EDuration="2.324937248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:44.322677026 +0000 UTC m=+3683.284227346" watchObservedRunningTime="2026-11-02 22:35:44.324937248 +0000 UTC m=+3683.286487602" Nov 02 22:35:45.216674 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:45.216639 4285 generic.go:356] "Generic (PLEG): container finished" podID="e223aabb-8121-452e-970e-9ab379b5d300" containerID="b42cb0c2be56631b1eb106b374d23a94b1a12aa1bc2649dd0d53c7cd42ce57a4" exitCode=0 Nov 02 22:35:45.217041 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:45.216707 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" event={"ID":"e223aabb-8121-452e-970e-9ab379b5d300","Type":"ContainerDied","Data":"b42cb0c2be56631b1eb106b374d23a94b1a12aa1bc2649dd0d53c7cd42ce57a4"} Nov 02 22:35:46.004396 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:35:46.004364 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:35:46.331188 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.331159 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:46.368467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.368443 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:46.556014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.555937 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca\") pod \"e223aabb-8121-452e-970e-9ab379b5d300\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " Nov 02 22:35:46.556200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.556033 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq\" (UniqueName: \"kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq\") pod \"e223aabb-8121-452e-970e-9ab379b5d300\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " Nov 02 22:35:46.556200 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.556162 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca\") pod \"e223aabb-8121-452e-970e-9ab379b5d300\" (UID: \"e223aabb-8121-452e-970e-9ab379b5d300\") " Nov 02 22:35:46.556530 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.556495 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca" pod "e223aabb-8121-452e-970e-9ab379b5d300" (UID: "e223aabb-8121-452e-970e-9ab379b5d300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:46.558095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.558068 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca" pod "e223aabb-8121-452e-970e-9ab379b5d300" (UID: "e223aabb-8121-452e-970e-9ab379b5d300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:46.558223 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.558203 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq" pod "e223aabb-8121-452e-970e-9ab379b5d300" (UID: "e223aabb-8121-452e-970e-9ab379b5d300"). InnerVolumeSpecName "kube-api-access-czkkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:46.657567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.657540 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:35:46.657567 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.657565 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkkq\" (UniqueName: \"kubernetes.io/projected/e223aabb-8121-452e-970e-9ab379b5d300-kube-api-access-czkkq\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:35:46.657715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:46.657576 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e223aabb-8121-452e-970e-9ab379b5d300-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:35:47.227897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:47.227869 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" Nov 02 22:35:47.227897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:47.227883 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb" event={"ID":"e223aabb-8121-452e-970e-9ab379b5d300","Type":"ContainerDied","Data":"73aebcb70814c2e31e3092bbc9ada3073b8bcd075af9d00b9a7ec4cfa1565450"} Nov 02 22:35:47.228102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:47.227924 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73aebcb70814c2e31e3092bbc9ada3073b8bcd075af9d00b9a7ec4cfa1565450" Nov 02 22:35:47.304069 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:47.304042 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:51.201457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:51.201415 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:54.146829 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:54.146795 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb"] Nov 02 22:35:54.250484 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:54.250454 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9"] Nov 02 22:35:54.533779 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:54.533751 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:54.739857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:54.739818 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:56.004388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:56.004351 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:59.147847 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:59.147810 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerName="payload-processing" containerID="cri-o://c8c6534b1b24d3ca96d0d5adc70c719a6a99c345824967ec21ce14c5b7f94109" gracePeriod=25 Nov 02 22:35:59.251019 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:35:59.250972 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerName="payload-pre-processing" containerID="cri-o://74627acbbf0e592ae8ada5c1b0cbeacbbccda10718b9245000be10e63b14472b" gracePeriod=25 Nov 02 22:36:00.163017 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:00.162986 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:01.693042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:01.693006 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:01.939983 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:01.939930 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:02.266640 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:02.266601 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb"] Nov 02 22:36:02.315987 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:02.315959 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8b2f0b-6k4cb"] Nov 02 22:36:02.709734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:02.709702 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:03.172886 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:03.172813 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e223aabb-8121-452e-970e-9ab379b5d300" path="/var/lib/kubelet/pods/e223aabb-8121-452e-970e-9ab379b5d300/volumes" Nov 02 22:36:03.741504 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:03.741472 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:36:03.741889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:03.741473 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:36:08.291297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:08.291262 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:08.474526 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:08.474488 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:08.636744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:08.636656 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc"] Nov 02 22:36:08.636942 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:08.636914 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" containerID="cri-o://698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d" gracePeriod=30 Nov 02 22:36:08.728786 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:08.728747 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:13.742241 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:13.742210 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:36:13.742568 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:13.742279 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:36:13.886193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:13.886170 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:36:14.011477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011451 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011480 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011504 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011548 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011622 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011591 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011622 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011690 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011754 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:14.011851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011779 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011815 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket\") pod \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\" (UID: \"a7a26cc0-6354-4fdd-8990-be2e270c4e4f\") " Nov 02 22:36:14.011851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.011765 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:14.012216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012059 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:14.012216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012093 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:14.012216 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012207 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.012388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012230 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.012388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012244 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.012388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012257 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.012388 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.012288 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:14.013766 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.013740 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:36:14.013897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.013880 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:14.014207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.014186 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "kube-api-access-78ndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:14.014207 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.014190 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy" pod "a7a26cc0-6354-4fdd-8990-be2e270c4e4f" (UID: "a7a26cc0-6354-4fdd-8990-be2e270c4e4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:14.113366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.113340 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.113366 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.113363 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.113551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.113374 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ndq\" (UniqueName: \"kubernetes.io/projected/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-kube-api-access-78ndq\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.113551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.113384 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.113551 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.113392 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a26cc0-6354-4fdd-8990-be2e270c4e4f-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:14.362446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.362351 4285 generic.go:356] "Generic (PLEG): container finished" podID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerID="698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d" exitCode=0 Nov 02 22:36:14.362446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.362394 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" Nov 02 22:36:14.362446 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.362433 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" event={"ID":"a7a26cc0-6354-4fdd-8990-be2e270c4e4f","Type":"ContainerDied","Data":"698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d"} Nov 02 22:36:14.362727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.362498 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc" event={"ID":"a7a26cc0-6354-4fdd-8990-be2e270c4e4f","Type":"ContainerDied","Data":"683229ef886ec23d26b5927984426a74b112e543d11f078435cee9acb69f0f60"} Nov 02 22:36:14.362727 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.362522 4285 scope.go:122] "RemoveContainer" containerID="698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d" Nov 02 22:36:14.372398 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.372382 4285 scope.go:122] "RemoveContainer" containerID="536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755" Nov 02 22:36:14.381072 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.381049 4285 scope.go:122] "RemoveContainer" containerID="698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d" Nov 02 22:36:14.381369 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:36:14.381350 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d\": container with ID starting with 698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d not found: ID does not exist" containerID="698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d" Nov 02 22:36:14.381466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.381376 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d"} err="failed to get container status \"698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d\": rpc error: code = NotFound desc = could not find container \"698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d\": container with ID starting with 698c8ee1c31b2c3ba4068e026cc11c62593a0400b96621a925dc26a1ba48011d not found: ID does not exist" Nov 02 22:36:14.381466 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.381392 4285 scope.go:122] "RemoveContainer" containerID="536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755" Nov 02 22:36:14.381644 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:36:14.381623 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755\": container with ID starting with 536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755 not found: ID does not exist" containerID="536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755" Nov 02 22:36:14.381709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.381649 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755"} err="failed to get container status \"536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755\": rpc error: code = NotFound desc = could not find container \"536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755\": container with ID starting with 536c792bbf25afdc076625e8f4aa253895a7c61bfa2b3ebeeb3643c889dd1755 not found: ID does not exist" Nov 02 22:36:14.527695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.527664 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc"] Nov 02 22:36:14.531105 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:14.531080 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8b2f0b-openshift-default-597c84bfdfbw6lc"] Nov 02 22:36:15.166561 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:15.166512 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:15.175367 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:15.175336 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" path="/var/lib/kubelet/pods/a7a26cc0-6354-4fdd-8990-be2e270c4e4f/volumes" Nov 02 22:36:16.004878 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:36:16.004847 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:36:19.387095 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.387070 4285 generic.go:356] "Generic (PLEG): container finished" podID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerID="c8c6534b1b24d3ca96d0d5adc70c719a6a99c345824967ec21ce14c5b7f94109" exitCode=0 Nov 02 22:36:19.387523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.387156 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" event={"ID":"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3","Type":"ContainerDied","Data":"c8c6534b1b24d3ca96d0d5adc70c719a6a99c345824967ec21ce14c5b7f94109"} Nov 02 22:36:19.387523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.387208 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" event={"ID":"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3","Type":"ContainerDied","Data":"81edd9f3de4cc20a60f4cba31a96c433d1f334de0ac073109422ac0056625157"} Nov 02 22:36:19.387523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.387225 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81edd9f3de4cc20a60f4cba31a96c433d1f334de0ac073109422ac0056625157" Nov 02 22:36:19.388840 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.388819 4285 generic.go:356] "Generic (PLEG): container finished" podID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerID="74627acbbf0e592ae8ada5c1b0cbeacbbccda10718b9245000be10e63b14472b" exitCode=0 Nov 02 22:36:19.388952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.388847 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" event={"ID":"0b8f2f90-e807-4afe-86d5-559279ee81be","Type":"ContainerDied","Data":"74627acbbf0e592ae8ada5c1b0cbeacbbccda10718b9245000be10e63b14472b"} Nov 02 22:36:19.401438 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.401418 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:36:19.462806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.462778 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume\") pod \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " Nov 02 22:36:19.462962 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.462835 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz\" (UniqueName: \"kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz\") pod \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\" (UID: \"9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3\") " Nov 02 22:36:19.463260 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.463230 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume" pod "9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" (UID: "9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:19.466467 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.466445 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz" pod "9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" (UID: "9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3"). InnerVolumeSpecName "kube-api-access-gd4rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:19.498043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.498017 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:36:19.564378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.564347 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume\") pod \"0b8f2f90-e807-4afe-86d5-559279ee81be\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " Nov 02 22:36:19.564536 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.564479 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m\" (UniqueName: \"kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m\") pod \"0b8f2f90-e807-4afe-86d5-559279ee81be\" (UID: \"0b8f2f90-e807-4afe-86d5-559279ee81be\") " Nov 02 22:36:19.564797 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.564769 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume" pod "0b8f2f90-e807-4afe-86d5-559279ee81be" (UID: "0b8f2f90-e807-4afe-86d5-559279ee81be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:19.564890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.564829 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:19.564890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.564852 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4rz\" (UniqueName: \"kubernetes.io/projected/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3-kube-api-access-gd4rz\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:19.566802 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.566773 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m" pod "0b8f2f90-e807-4afe-86d5-559279ee81be" (UID: "0b8f2f90-e807-4afe-86d5-559279ee81be"). InnerVolumeSpecName "kube-api-access-4578m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:19.666024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.666002 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8f2f90-e807-4afe-86d5-559279ee81be-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:19.666024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:19.666025 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4578m\" (UniqueName: \"kubernetes.io/projected/0b8f2f90-e807-4afe-86d5-559279ee81be-kube-api-access-4578m\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:36:20.397573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.397529 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" event={"ID":"0b8f2f90-e807-4afe-86d5-559279ee81be","Type":"ContainerDied","Data":"38894f1240a314c63903e00b619fa2df5176452a1d2a4f793b65c6230a029b91"} Nov 02 22:36:20.397573 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.397562 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9" Nov 02 22:36:20.398224 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.397606 4285 scope.go:122] "RemoveContainer" containerID="74627acbbf0e592ae8ada5c1b0cbeacbbccda10718b9245000be10e63b14472b" Nov 02 22:36:20.398224 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.397571 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb" Nov 02 22:36:20.563237 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.563193 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb"] Nov 02 22:36:20.612416 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.612381 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8b2f0b-69cb974dbcz6fb"] Nov 02 22:36:20.677142 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.677097 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9"] Nov 02 22:36:20.683469 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:20.683436 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8b2f0b-6ccdb2qxq9"] Nov 02 22:36:21.193424 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:21.193391 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" path="/var/lib/kubelet/pods/0b8f2f90-e807-4afe-86d5-559279ee81be/volumes" Nov 02 22:36:21.193748 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:21.193733 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" path="/var/lib/kubelet/pods/9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3/volumes" Nov 02 22:36:23.112247 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:23.112215 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:23.371190 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:23.371101 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:24.750436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:24.750409 4285 scope.go:122] "RemoveContainer" containerID="0e78f0246c5490d61139ae1f46639dee57abce92b46f6c1f0ba577ce8923dbef" Nov 02 22:36:24.947668 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:24.947634 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:39.209589 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:39.209507 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:46.004901 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:36:46.004858 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:36:48.047889 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:48.047855 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84"] Nov 02 22:36:48.096915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:48.096887 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tcx84"] Nov 02 22:36:49.171588 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:49.171555 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8fdaae-67a2-47b9-9581-061212594a97" path="/var/lib/kubelet/pods/4c8fdaae-67a2-47b9-9581-061212594a97/volumes" Nov 02 22:36:57.573679 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:36:57.573642 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:13.025301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:13.025270 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:16.004501 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:37:16.004463 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:37:23.424082 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:23.424037 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:31.611893 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.611856 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd"] Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612249 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612259 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612273 4285 cpu_manager.go:414] "containerMap: removing container" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612278 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612286 4285 cpu_manager.go:414] "containerMap: removing container" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerName="payload-processing" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612291 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerName="payload-processing" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612303 4285 cpu_manager.go:414] "containerMap: removing container" podUID="e223aabb-8121-452e-970e-9ab379b5d300" containerName="revoke-keys" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612307 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e223aabb-8121-452e-970e-9ab379b5d300" containerName="revoke-keys" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612315 4285 cpu_manager.go:414] "containerMap: removing container" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerName="payload-pre-processing" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612320 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerName="payload-pre-processing" Nov 02 22:37:31.612376 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612377 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" Nov 02 22:37:31.612717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612387 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a26cc0-6354-4fdd-8990-be2e270c4e4f" containerName="istio-proxy" Nov 02 22:37:31.612717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612399 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6f8fa7-4bf8-4915-b01f-b18f9fcd60c3" containerName="payload-processing" Nov 02 22:37:31.612717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612409 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="e223aabb-8121-452e-970e-9ab379b5d300" containerName="revoke-keys" Nov 02 22:37:31.612717 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.612415 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8f2f90-e807-4afe-86d5-559279ee81be" containerName="payload-pre-processing" Nov 02 22:37:31.615381 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.615366 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.718313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.718291 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:31.718313 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.718311 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:37:31.720347 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.720329 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd"] Nov 02 22:37:31.732031 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.732005 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.732176 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.732159 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7s2n\" (UniqueName: \"kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.732225 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.732182 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.833496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.833465 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.833629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.833575 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7s2n\" (UniqueName: \"kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.833629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.833604 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.834242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.834221 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.835739 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.835723 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.891718 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.891662 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7s2n\" (UniqueName: \"kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n\") pod \"maas-api-revoke-sub-452b6d7b-2ztpd\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:31.927497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:31.927472 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:32.296904 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:32.296883 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd"] Nov 02 22:37:32.298731 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:37:32.298702 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd63805_8355_4807_89da_f9e8b6f4cba1.slice/crio-7eba337f46ad329913b3981f8f74d571ec3f0bb2cca6f55cdb094829eac17772 WatchSource:0}: Error finding container 7eba337f46ad329913b3981f8f74d571ec3f0bb2cca6f55cdb094829eac17772: Status 404 returned error can't find the container with id 7eba337f46ad329913b3981f8f74d571ec3f0bb2cca6f55cdb094829eac17772 Nov 02 22:37:32.777456 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:32.777425 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" event={"ID":"dbd63805-8355-4807-89da-f9e8b6f4cba1","Type":"ContainerStarted","Data":"0ad9cbe0068ca067e3f1e8f59b1c224057db031b1c53292a9e445767018dc5f4"} Nov 02 22:37:32.777881 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:32.777479 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" event={"ID":"dbd63805-8355-4807-89da-f9e8b6f4cba1","Type":"ContainerStarted","Data":"7eba337f46ad329913b3981f8f74d571ec3f0bb2cca6f55cdb094829eac17772"} Nov 02 22:37:32.891308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:32.891229 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" podStartSLOduration=1.891215581 podStartE2EDuration="1.891215581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:32.889742279 +0000 UTC m=+3791.851292597" watchObservedRunningTime="2026-11-02 22:37:32.891215581 +0000 UTC m=+3791.852765896" Nov 02 22:37:33.785477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:33.785443 4285 generic.go:356] "Generic (PLEG): container finished" podID="dbd63805-8355-4807-89da-f9e8b6f4cba1" containerID="0ad9cbe0068ca067e3f1e8f59b1c224057db031b1c53292a9e445767018dc5f4" exitCode=0 Nov 02 22:37:33.785477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:33.785483 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" event={"ID":"dbd63805-8355-4807-89da-f9e8b6f4cba1","Type":"ContainerDied","Data":"0ad9cbe0068ca067e3f1e8f59b1c224057db031b1c53292a9e445767018dc5f4"} Nov 02 22:37:35.125159 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.124824 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:35.300171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.299582 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n\" (UniqueName: \"kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n\") pod \"dbd63805-8355-4807-89da-f9e8b6f4cba1\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " Nov 02 22:37:35.300171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.299635 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca\") pod \"dbd63805-8355-4807-89da-f9e8b6f4cba1\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " Nov 02 22:37:35.300171 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.299655 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca\") pod \"dbd63805-8355-4807-89da-f9e8b6f4cba1\" (UID: \"dbd63805-8355-4807-89da-f9e8b6f4cba1\") " Nov 02 22:37:35.300539 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.300356 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca" pod "dbd63805-8355-4807-89da-f9e8b6f4cba1" (UID: "dbd63805-8355-4807-89da-f9e8b6f4cba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:37:35.303574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.303172 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca" pod "dbd63805-8355-4807-89da-f9e8b6f4cba1" (UID: "dbd63805-8355-4807-89da-f9e8b6f4cba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:37:35.303574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.303231 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n" pod "dbd63805-8355-4807-89da-f9e8b6f4cba1" (UID: "dbd63805-8355-4807-89da-f9e8b6f4cba1"). InnerVolumeSpecName "kube-api-access-t7s2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:37:35.400922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.400794 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7s2n\" (UniqueName: \"kubernetes.io/projected/dbd63805-8355-4807-89da-f9e8b6f4cba1-kube-api-access-t7s2n\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:35.400922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.400831 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:35.400922 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.400846 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbd63805-8355-4807-89da-f9e8b6f4cba1-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:37:35.798872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.798836 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" event={"ID":"dbd63805-8355-4807-89da-f9e8b6f4cba1","Type":"ContainerDied","Data":"7eba337f46ad329913b3981f8f74d571ec3f0bb2cca6f55cdb094829eac17772"} Nov 02 22:37:35.799043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.798885 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eba337f46ad329913b3981f8f74d571ec3f0bb2cca6f55cdb094829eac17772" Nov 02 22:37:35.799043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:35.798849 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd" Nov 02 22:37:41.087843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:41.087816 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:44.447497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:44.447462 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:46.004237 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:37:46.004205 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:37:54.961246 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:54.961215 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:59.085264 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.085229 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5"] Nov 02 22:37:59.085687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.085670 4285 cpu_manager.go:414] "containerMap: removing container" podUID="dbd63805-8355-4807-89da-f9e8b6f4cba1" containerName="revoke-keys" Nov 02 22:37:59.085687 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.085687 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd63805-8355-4807-89da-f9e8b6f4cba1" containerName="revoke-keys" Nov 02 22:37:59.085792 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.085765 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd63805-8355-4807-89da-f9e8b6f4cba1" containerName="revoke-keys" Nov 02 22:37:59.088675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.088659 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.128151 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.128092 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knffv\" (UniqueName: \"kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.128325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.128237 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.128325 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.128272 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.193409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.193383 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:59.193409 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.193392 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:37:59.195659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.195637 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5"] Nov 02 22:37:59.229184 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.229157 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knffv\" (UniqueName: \"kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.229324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.229227 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.229324 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.229274 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.230167 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.230145 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.231635 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.231611 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.289320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.289297 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knffv\" (UniqueName: \"kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv\") pod \"maas-api-revoke-sub-3e31a506-czxq5\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.400951 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.400870 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:37:59.770223 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.770199 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5"] Nov 02 22:37:59.772243 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:37:59.772214 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595aec09_dc34_49b8_834f_1821f988548f.slice/crio-8e6fdf517484b65408df894de252a3fe6cf05f3fa42670937b6a191590731a01 WatchSource:0}: Error finding container 8e6fdf517484b65408df894de252a3fe6cf05f3fa42670937b6a191590731a01: Status 404 returned error can't find the container with id 8e6fdf517484b65408df894de252a3fe6cf05f3fa42670937b6a191590731a01 Nov 02 22:37:59.930458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:37:59.930393 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" event={"ID":"595aec09-dc34-49b8-834f-1821f988548f","Type":"ContainerStarted","Data":"8e6fdf517484b65408df894de252a3fe6cf05f3fa42670937b6a191590731a01"} Nov 02 22:38:00.968147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:00.963886 4285 generic.go:356] "Generic (PLEG): container finished" podID="595aec09-dc34-49b8-834f-1821f988548f" containerID="ec64cdf3b6b9c17ddee261c0136bff67fbb3bf91bad552675cc407e08fed7a0d" exitCode=0 Nov 02 22:38:00.968147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:00.964015 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" event={"ID":"595aec09-dc34-49b8-834f-1821f988548f","Type":"ContainerDied","Data":"ec64cdf3b6b9c17ddee261c0136bff67fbb3bf91bad552675cc407e08fed7a0d"} Nov 02 22:38:02.315474 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.315442 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:38:02.397059 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.397028 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca\") pod \"595aec09-dc34-49b8-834f-1821f988548f\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " Nov 02 22:38:02.397248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.397066 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca\") pod \"595aec09-dc34-49b8-834f-1821f988548f\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " Nov 02 22:38:02.397248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.397152 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv\" (UniqueName: \"kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv\") pod \"595aec09-dc34-49b8-834f-1821f988548f\" (UID: \"595aec09-dc34-49b8-834f-1821f988548f\") " Nov 02 22:38:02.397490 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.397464 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca" pod "595aec09-dc34-49b8-834f-1821f988548f" (UID: "595aec09-dc34-49b8-834f-1821f988548f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:02.397629 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.397610 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/595aec09-dc34-49b8-834f-1821f988548f-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.399215 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.399189 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca" pod "595aec09-dc34-49b8-834f-1821f988548f" (UID: "595aec09-dc34-49b8-834f-1821f988548f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:02.399322 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.399269 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv" pod "595aec09-dc34-49b8-834f-1821f988548f" (UID: "595aec09-dc34-49b8-834f-1821f988548f"). InnerVolumeSpecName "kube-api-access-knffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:02.498641 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.498616 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/595aec09-dc34-49b8-834f-1821f988548f-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.498641 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.498641 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knffv\" (UniqueName: \"kubernetes.io/projected/595aec09-dc34-49b8-834f-1821f988548f-kube-api-access-knffv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.983846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.983819 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" Nov 02 22:38:02.983846 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.983829 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5" event={"ID":"595aec09-dc34-49b8-834f-1821f988548f","Type":"ContainerDied","Data":"8e6fdf517484b65408df894de252a3fe6cf05f3fa42670937b6a191590731a01"} Nov 02 22:38:02.984143 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:02.983872 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e6fdf517484b65408df894de252a3fe6cf05f3fa42670937b6a191590731a01" Nov 02 22:38:16.005500 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:38:16.005421 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:38:23.793862 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:23.793826 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:24.783760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:24.783733 4285 scope.go:122] "RemoveContainer" containerID="1eaf5d840a910dbc1e64204e896af8f6c9962dd47733726d0ceeac04bc0172af" Nov 02 22:38:29.429273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.429240 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df"] Nov 02 22:38:29.429670 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.429657 4285 cpu_manager.go:414] "containerMap: removing container" podUID="595aec09-dc34-49b8-834f-1821f988548f" containerName="revoke-keys" Nov 02 22:38:29.429714 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.429671 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595aec09-dc34-49b8-834f-1821f988548f" containerName="revoke-keys" Nov 02 22:38:29.429791 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.429781 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="595aec09-dc34-49b8-834f-1821f988548f" containerName="revoke-keys" Nov 02 22:38:29.432737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.432715 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.486299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.486245 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:29.535951 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.535917 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c4p9p\"" Nov 02 22:38:29.538219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.538194 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df"] Nov 02 22:38:29.563320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.563290 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.563459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.563328 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrzf\" (UniqueName: \"kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.563554 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.563538 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.664364 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.664335 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.664497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.664383 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.664497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.664410 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrzf\" (UniqueName: \"kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.665004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.664985 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.666664 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.666644 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.720506 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.720485 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrzf\" (UniqueName: \"kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf\") pod \"maas-api-revoke-sub-852dca62-fk9df\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:29.745018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:29.744996 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:30.114501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:30.114474 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df"] Nov 02 22:38:30.116309 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:38:30.116283 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d3e40a_8ac7_42ec_933c_537d1673bdfe.slice/crio-3181675f82c2f78a95644f405bb17e14fdc31737819788cbf3bdf674ee4f0a8f WatchSource:0}: Error finding container 3181675f82c2f78a95644f405bb17e14fdc31737819788cbf3bdf674ee4f0a8f: Status 404 returned error can't find the container with id 3181675f82c2f78a95644f405bb17e14fdc31737819788cbf3bdf674ee4f0a8f Nov 02 22:38:30.159178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:30.159150 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" event={"ID":"38d3e40a-8ac7-42ec-933c-537d1673bdfe","Type":"ContainerStarted","Data":"3181675f82c2f78a95644f405bb17e14fdc31737819788cbf3bdf674ee4f0a8f"} Nov 02 22:38:31.166403 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:31.166324 4285 generic.go:356] "Generic (PLEG): container finished" podID="38d3e40a-8ac7-42ec-933c-537d1673bdfe" containerID="ef43ca3c56dc1e5597bb5eb73125e43ec483f9193ec8e9c8f088d93a0a1fb03c" exitCode=0 Nov 02 22:38:31.171442 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:31.171414 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" event={"ID":"38d3e40a-8ac7-42ec-933c-537d1673bdfe","Type":"ContainerDied","Data":"ef43ca3c56dc1e5597bb5eb73125e43ec483f9193ec8e9c8f088d93a0a1fb03c"} Nov 02 22:38:32.298898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.298876 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:32.391583 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.391560 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca\") pod \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " Nov 02 22:38:32.391716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.391599 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca\") pod \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " Nov 02 22:38:32.391716 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.391660 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf\" (UniqueName: \"kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf\") pod \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\" (UID: \"38d3e40a-8ac7-42ec-933c-537d1673bdfe\") " Nov 02 22:38:32.391994 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.391971 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca" pod "38d3e40a-8ac7-42ec-933c-537d1673bdfe" (UID: "38d3e40a-8ac7-42ec-933c-537d1673bdfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:32.393556 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.393531 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca" pod "38d3e40a-8ac7-42ec-933c-537d1673bdfe" (UID: "38d3e40a-8ac7-42ec-933c-537d1673bdfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:32.393715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.393692 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf" pod "38d3e40a-8ac7-42ec-933c-537d1673bdfe" (UID: "38d3e40a-8ac7-42ec-933c-537d1673bdfe"). InnerVolumeSpecName "kube-api-access-vjrzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:32.492602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.492576 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-xks-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:32.492602 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.492600 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d3e40a-8ac7-42ec-933c-537d1673bdfe-maas-api-openshift-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:32.492768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:32.492610 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrzf\" (UniqueName: \"kubernetes.io/projected/38d3e40a-8ac7-42ec-933c-537d1673bdfe-kube-api-access-vjrzf\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:33.176362 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:33.176331 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" event={"ID":"38d3e40a-8ac7-42ec-933c-537d1673bdfe","Type":"ContainerDied","Data":"3181675f82c2f78a95644f405bb17e14fdc31737819788cbf3bdf674ee4f0a8f"} Nov 02 22:38:33.176546 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:33.176364 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df" Nov 02 22:38:33.176546 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:33.176373 4285 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3181675f82c2f78a95644f405bb17e14fdc31737819788cbf3bdf674ee4f0a8f" Nov 02 22:38:35.044982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:35.044945 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f"] Nov 02 22:38:35.094193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:35.094165 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-ft62f"] Nov 02 22:38:35.171504 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:35.171474 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1" path="/var/lib/kubelet/pods/fbeeb90a-41a2-48f4-ac46-703dc3ebe4f1/volumes" Nov 02 22:38:42.518462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.518426 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc"] Nov 02 22:38:42.519055 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.519035 4285 cpu_manager.go:414] "containerMap: removing container" podUID="38d3e40a-8ac7-42ec-933c-537d1673bdfe" containerName="revoke-keys" Nov 02 22:38:42.519185 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.519058 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d3e40a-8ac7-42ec-933c-537d1673bdfe" containerName="revoke-keys" Nov 02 22:38:42.519263 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.519195 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d3e40a-8ac7-42ec-933c-537d1673bdfe" containerName="revoke-keys" Nov 02 22:38:42.523544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.523508 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg"] Nov 02 22:38:42.523779 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.523757 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.527102 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.527085 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.558076 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.558050 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc"] Nov 02 22:38:42.604318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.604277 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:38:42.604318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.604277 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 22:38:42.604584 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.604361 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cbb6p\"" Nov 02 22:38:42.627965 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.627934 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg"] Nov 02 22:38:42.687387 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687358 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhn8g\" (UniqueName: \"kubernetes.io/projected/e4a6cacc-a52e-4add-8189-67f200ca638a-kube-api-access-hhn8g\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.687544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687399 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a6cacc-a52e-4add-8189-67f200ca638a-metrics-certs\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.687544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687424 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.687544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687517 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a6cacc-a52e-4add-8189-67f200ca638a-openshift-service-ca\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.687659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687549 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4a6cacc-a52e-4add-8189-67f200ca638a-maas-api-tls\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.687659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687644 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgzs\" (UniqueName: \"kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.687729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687683 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.687729 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.687713 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.789042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.788979 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.789042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789011 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.789273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789045 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhn8g\" (UniqueName: \"kubernetes.io/projected/e4a6cacc-a52e-4add-8189-67f200ca638a-kube-api-access-hhn8g\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.789273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789158 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a6cacc-a52e-4add-8189-67f200ca638a-metrics-certs\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.789273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789209 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.789273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789244 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a6cacc-a52e-4add-8189-67f200ca638a-openshift-service-ca\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.789477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789273 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4a6cacc-a52e-4add-8189-67f200ca638a-maas-api-tls\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.789477 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789346 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgzs\" (UniqueName: \"kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.789927 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.789903 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.790147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.790111 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a6cacc-a52e-4add-8189-67f200ca638a-openshift-service-ca\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.791544 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.791514 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4a6cacc-a52e-4add-8189-67f200ca638a-metrics-certs\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.791634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.791578 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.791834 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.791817 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4a6cacc-a52e-4add-8189-67f200ca638a-maas-api-tls\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.791872 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.791831 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.844728 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.844706 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgzs\" (UniqueName: \"kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs\") pod \"maas-api-57cf7b58d8-ht7lc\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:42.844817 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.844747 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhn8g\" (UniqueName: \"kubernetes.io/projected/e4a6cacc-a52e-4add-8189-67f200ca638a-kube-api-access-hhn8g\") pod \"maas-api-54b8648dd4-jfblg\" (UID: \"e4a6cacc-a52e-4add-8189-67f200ca638a\") " pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:42.848589 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:42.848575 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:43.139706 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:43.139623 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:43.220672 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:43.220644 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg"] Nov 02 22:38:43.225659 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:43.225638 4285 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:38:43.514003 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:43.513981 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc"] Nov 02 22:38:43.516006 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:38:43.515980 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7184cca_b7c9_4bf2_b432_3bb0814af854.slice/crio-5a6755739a043513f916d78f0b89cc9a9d989aeaf7aab9864075fd1436bb0855 WatchSource:0}: Error finding container 5a6755739a043513f916d78f0b89cc9a9d989aeaf7aab9864075fd1436bb0855: Status 404 returned error can't find the container with id 5a6755739a043513f916d78f0b89cc9a9d989aeaf7aab9864075fd1436bb0855 Nov 02 22:38:44.233837 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:44.233813 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" event={"ID":"c7184cca-b7c9-4bf2-b432-3bb0814af854","Type":"ContainerStarted","Data":"5a6755739a043513f916d78f0b89cc9a9d989aeaf7aab9864075fd1436bb0855"} Nov 02 22:38:44.235276 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:44.235254 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" event={"ID":"e4a6cacc-a52e-4add-8189-67f200ca638a","Type":"ContainerStarted","Data":"3c08ff3bf9590f2ccdc6c82ff3490de3e32b2f4eb1abc551bbad75008e8e68ee"} Nov 02 22:38:44.235399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:44.235301 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" event={"ID":"e4a6cacc-a52e-4add-8189-67f200ca638a","Type":"ContainerStarted","Data":"6af288b17b01176b173a229f57b45255bc40406dc396599bec16d241330bd28c"} Nov 02 22:38:44.235462 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:44.235405 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:44.357081 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:44.357032 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" podStartSLOduration=1.415827996 podStartE2EDuration="2.357019912s" totalImagesPullingTime="941.191916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:44.355164462 +0000 UTC m=+3863.316714780" watchObservedRunningTime="2026-11-02 22:38:44.357019912 +0000 UTC m=+3863.318570230" Nov 02 22:38:45.241763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:45.241724 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" event={"ID":"c7184cca-b7c9-4bf2-b432-3bb0814af854","Type":"ContainerStarted","Data":"cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e"} Nov 02 22:38:45.242413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:45.241826 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:45.355979 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:45.355929 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" podStartSLOduration=2.463081524 podStartE2EDuration="3.355916515s" totalImagesPullingTime="892.834991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:45.354763643 +0000 UTC m=+3864.316313962" watchObservedRunningTime="2026-11-02 22:38:45.355916515 +0000 UTC m=+3864.317466833" Nov 02 22:38:46.004563 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:38:46.004528 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:38:48.538266 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:48.538232 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:49.074395 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:49.074360 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:49.671461 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:49.671427 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:49.893665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:49.893636 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:50.248013 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.247986 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b8648dd4-jfblg" Nov 02 22:38:50.417763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.417728 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc"] Nov 02 22:38:50.424958 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.418571 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" podUID="c7184cca-b7c9-4bf2-b432-3bb0814af854" containerName="maas-api" containerID="cri-o://cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e" gracePeriod=30 Nov 02 22:38:50.432301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.432277 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:50.671826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.671803 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:50.763564 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.763537 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs\") pod \"c7184cca-b7c9-4bf2-b432-3bb0814af854\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " Nov 02 22:38:50.763694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.763574 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs\" (UniqueName: \"kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs\") pod \"c7184cca-b7c9-4bf2-b432-3bb0814af854\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " Nov 02 22:38:50.763763 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.763705 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls\") pod \"c7184cca-b7c9-4bf2-b432-3bb0814af854\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " Nov 02 22:38:50.763826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.763792 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca\") pod \"c7184cca-b7c9-4bf2-b432-3bb0814af854\" (UID: \"c7184cca-b7c9-4bf2-b432-3bb0814af854\") " Nov 02 22:38:50.764178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.764151 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca" pod "c7184cca-b7c9-4bf2-b432-3bb0814af854" (UID: "c7184cca-b7c9-4bf2-b432-3bb0814af854"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:50.765525 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.765497 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs" pod "c7184cca-b7c9-4bf2-b432-3bb0814af854" (UID: "c7184cca-b7c9-4bf2-b432-3bb0814af854"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:50.765685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.765670 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs" pod "c7184cca-b7c9-4bf2-b432-3bb0814af854" (UID: "c7184cca-b7c9-4bf2-b432-3bb0814af854"). InnerVolumeSpecName "kube-api-access-wpgzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:50.765736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.765695 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls" pod "c7184cca-b7c9-4bf2-b432-3bb0814af854" (UID: "c7184cca-b7c9-4bf2-b432-3bb0814af854"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:50.864737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.864670 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:50.864737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.864698 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7184cca-b7c9-4bf2-b432-3bb0814af854-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:50.864737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.864707 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7184cca-b7c9-4bf2-b432-3bb0814af854-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:50.864737 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:50.864715 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgzs\" (UniqueName: \"kubernetes.io/projected/c7184cca-b7c9-4bf2-b432-3bb0814af854-kube-api-access-wpgzs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:38:51.271521 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.271488 4285 generic.go:356] "Generic (PLEG): container finished" podID="c7184cca-b7c9-4bf2-b432-3bb0814af854" containerID="cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e" exitCode=0 Nov 02 22:38:51.271675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.271530 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" Nov 02 22:38:51.271675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.271571 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" event={"ID":"c7184cca-b7c9-4bf2-b432-3bb0814af854","Type":"ContainerDied","Data":"cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e"} Nov 02 22:38:51.271675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.271620 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc" event={"ID":"c7184cca-b7c9-4bf2-b432-3bb0814af854","Type":"ContainerDied","Data":"5a6755739a043513f916d78f0b89cc9a9d989aeaf7aab9864075fd1436bb0855"} Nov 02 22:38:51.271675 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.271655 4285 scope.go:122] "RemoveContainer" containerID="cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e" Nov 02 22:38:51.280191 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.280170 4285 scope.go:122] "RemoveContainer" containerID="cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e" Nov 02 22:38:51.280449 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:38:51.280432 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e\": container with ID starting with cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e not found: ID does not exist" containerID="cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e" Nov 02 22:38:51.280513 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.280458 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e"} err="failed to get container status \"cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e\": rpc error: code = NotFound desc = could not find container \"cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e\": container with ID starting with cabdc3d5d946f50dc5699aa7e97698d6c1be5a72cfd10867d0447cccb959c37e not found: ID does not exist" Nov 02 22:38:51.435164 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.435072 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc"] Nov 02 22:38:51.484465 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:51.484435 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cf7b58d8-ht7lc"] Nov 02 22:38:53.171511 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:53.171476 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7184cca-b7c9-4bf2-b432-3bb0814af854" path="/var/lib/kubelet/pods/c7184cca-b7c9-4bf2-b432-3bb0814af854/volumes" Nov 02 22:38:53.552196 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:53.552161 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:53.782290 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:53.782262 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:56.854901 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:56.854865 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:57.024843 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.024810 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:57.225332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.225301 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt"] Nov 02 22:38:57.225702 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.225689 4285 cpu_manager.go:414] "containerMap: removing container" podUID="c7184cca-b7c9-4bf2-b432-3bb0814af854" containerName="maas-api" Nov 02 22:38:57.225749 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.225703 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7184cca-b7c9-4bf2-b432-3bb0814af854" containerName="maas-api" Nov 02 22:38:57.225794 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.225766 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7184cca-b7c9-4bf2-b432-3bb0814af854" containerName="maas-api" Nov 02 22:38:57.230174 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.230159 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.286840 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.286816 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ce65d173-dockercfg-b4rhn\"" Nov 02 22:38:57.286954 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.286847 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ce65d173\"" Nov 02 22:38:57.317896 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.317866 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.318254 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.318225 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmcq\" (UniqueName: \"kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq\") pod \"payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.340545 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.340512 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt"] Nov 02 22:38:57.356646 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.356611 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh"] Nov 02 22:38:57.361636 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.361606 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.419335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.419303 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmcq\" (UniqueName: \"kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq\") pod \"payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.419495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.419364 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.419495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.419410 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwxv\" (UniqueName: \"kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv\") pod \"payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.419495 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.419462 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.420215 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.420192 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.422609 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.422581 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh"] Nov 02 22:38:57.477598 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.477538 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmcq\" (UniqueName: \"kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq\") pod \"payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.519925 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.519900 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwxv\" (UniqueName: \"kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv\") pod \"payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.520077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.519954 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.520608 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.520588 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.542671 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.542648 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:57.577917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.577891 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwxv\" (UniqueName: \"kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv\") pod \"payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.676709 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.676676 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:57.925775 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.925751 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt"] Nov 02 22:38:57.927936 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:38:57.927901 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92648963_464b_4392_b018_f7769cd7e537.slice/crio-ebb929d58ca456435737cd3b36c8c5594a7bdc1c5130a6255511d3203ea00e9b WatchSource:0}: Error finding container ebb929d58ca456435737cd3b36c8c5594a7bdc1c5130a6255511d3203ea00e9b: Status 404 returned error can't find the container with id ebb929d58ca456435737cd3b36c8c5594a7bdc1c5130a6255511d3203ea00e9b Nov 02 22:38:57.941012 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.940990 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h"] Nov 02 22:38:57.946318 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:57.946300 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.004932 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.004910 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ce65d173-serving-cert\"" Nov 02 22:38:58.013849 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.013819 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h"] Nov 02 22:38:58.025285 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.025258 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s9c\" (UniqueName: \"kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.025436 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.025412 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.025517 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.025496 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.025586 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.025575 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.075083 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.075057 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh"] Nov 02 22:38:58.077423 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:38:58.077399 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b699042_b83a_46b8_a7cb_74df8367974d.slice/crio-b4f10be5db673e873e503f954c62958e20b66fb4261bd0cc8a0a5d4fa6e9a331 WatchSource:0}: Error finding container b4f10be5db673e873e503f954c62958e20b66fb4261bd0cc8a0a5d4fa6e9a331: Status 404 returned error can't find the container with id b4f10be5db673e873e503f954c62958e20b66fb4261bd0cc8a0a5d4fa6e9a331 Nov 02 22:38:58.126581 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.126557 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.126686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.126590 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.126686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.126625 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.126686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.126669 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s9c\" (UniqueName: \"kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.126862 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:38:58.126777 4285 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-serving-cert: secret "maas-api-e2e-ait-ce65d173-serving-cert" not found Nov 02 22:38:58.126862 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:38:58.126846 4285 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls podName:1981d04e-0e92-4985-ab50-ae0046c65ec3 nodeName:}" failed. No retries permitted until 2026-11-02 22:38:58.626824533 +0000 UTC m=+3877.588374850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls") pod "maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" (UID: "1981d04e-0e92-4985-ab50-ae0046c65ec3") : secret "maas-api-e2e-ait-ce65d173-serving-cert" not found Nov 02 22:38:58.127270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.127249 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.128798 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.128777 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.181328 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.181241 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:58.185528 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.185495 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s9c\" (UniqueName: \"kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.310997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.310911 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" event={"ID":"2b699042-b83a-46b8-a7cb-74df8367974d","Type":"ContainerStarted","Data":"67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90"} Nov 02 22:38:58.310997 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.310976 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" event={"ID":"2b699042-b83a-46b8-a7cb-74df8367974d","Type":"ContainerStarted","Data":"b4f10be5db673e873e503f954c62958e20b66fb4261bd0cc8a0a5d4fa6e9a331"} Nov 02 22:38:58.311259 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.311120 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:38:58.312464 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.312437 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" event={"ID":"92648963-464b-4392-b018-f7769cd7e537","Type":"ContainerStarted","Data":"64438430f2113ac8eec7942d6a51b02e8789758b51bcd731b7d861d6335a7e90"} Nov 02 22:38:58.312572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.312472 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" event={"ID":"92648963-464b-4392-b018-f7769cd7e537","Type":"ContainerStarted","Data":"ebb929d58ca456435737cd3b36c8c5594a7bdc1c5130a6255511d3203ea00e9b"} Nov 02 22:38:58.312572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.312561 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:38:58.358606 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.358574 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:58.429456 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.429305 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" podStartSLOduration=1.429288146 podStartE2EDuration="1.429288146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:58.427732502 +0000 UTC m=+3877.389282822" watchObservedRunningTime="2026-11-02 22:38:58.429288146 +0000 UTC m=+3877.390838489" Nov 02 22:38:58.521382 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.521352 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:58.553870 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.553807 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" podStartSLOduration=1.553787914 podStartE2EDuration="1.553787914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:58.550475378 +0000 UTC m=+3877.512025698" watchObservedRunningTime="2026-11-02 22:38:58.553787914 +0000 UTC m=+3877.515338233" Nov 02 22:38:58.641458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.641377 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.643743 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.643716 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\") pod \"maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:58.872941 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:58.872914 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:38:59.247654 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:59.247627 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h"] Nov 02 22:38:59.248977 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:38:59.248946 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1981d04e_0e92_4985_ab50_ae0046c65ec3.slice/crio-f7e76b2728f26b38b1b86e5ff98a77d03b9d67adabab4f4e50a8c787f54fd6b8 WatchSource:0}: Error finding container f7e76b2728f26b38b1b86e5ff98a77d03b9d67adabab4f4e50a8c787f54fd6b8: Status 404 returned error can't find the container with id f7e76b2728f26b38b1b86e5ff98a77d03b9d67adabab4f4e50a8c787f54fd6b8 Nov 02 22:38:59.318759 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:38:59.318733 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" event={"ID":"1981d04e-0e92-4985-ab50-ae0046c65ec3","Type":"ContainerStarted","Data":"f7e76b2728f26b38b1b86e5ff98a77d03b9d67adabab4f4e50a8c787f54fd6b8"} Nov 02 22:39:00.326628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:00.326543 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" event={"ID":"1981d04e-0e92-4985-ab50-ae0046c65ec3","Type":"ContainerStarted","Data":"3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10"} Nov 02 22:39:00.326628 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:00.326602 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:39:03.434811 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:03.434780 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:04.045695 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:04.045640 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" podStartSLOduration=6.238724224 podStartE2EDuration="7.045624303s" totalImagesPullingTime="806.900079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:00.47224176 +0000 UTC m=+3879.433792080" watchObservedRunningTime="2026-11-02 22:39:04.045624303 +0000 UTC m=+3883.007174647" Nov 02 22:39:04.045869 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:04.045812 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh"] Nov 02 22:39:04.095360 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:04.095326 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-vkjsh"] Nov 02 22:39:05.172375 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:05.172340 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cd755d-eb95-469c-a709-681d3c32642f" path="/var/lib/kubelet/pods/44cd755d-eb95-469c-a709-681d3c32642f/volumes" Nov 02 22:39:06.338862 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:06.338828 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:39:08.325027 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:08.324995 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:09.322458 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:09.322433 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:39:09.322641 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:09.322491 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:39:13.149310 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:13.149273 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:16.005023 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:39:16.004990 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:39:18.258061 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:18.258022 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:23.341985 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:23.341944 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:24.100917 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:24.100878 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:28.413479 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:28.413445 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:33.075982 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:33.075942 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:37.913739 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:37.913708 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:41.224952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:41.224919 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh"] Nov 02 22:39:41.332724 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:41.332695 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt"] Nov 02 22:39:41.620024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:41.619948 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:41.778591 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:41.778557 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:42.620509 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:42.620478 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:46.004207 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:39:46.004171 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:39:46.225645 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.225590 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" podUID="2b699042-b83a-46b8-a7cb-74df8367974d" containerName="payload-processing" containerID="cri-o://67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90" gracePeriod=25 Nov 02 22:39:46.333691 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.333635 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" podUID="92648963-464b-4392-b018-f7769cd7e537" containerName="payload-pre-processing" containerID="cri-o://64438430f2113ac8eec7942d6a51b02e8789758b51bcd731b7d861d6335a7e90" gracePeriod=25 Nov 02 22:39:46.478966 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.478938 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:39:46.500074 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.500046 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume\") pod \"2b699042-b83a-46b8-a7cb-74df8367974d\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " Nov 02 22:39:46.500262 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.500083 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv\" (UniqueName: \"kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv\") pod \"2b699042-b83a-46b8-a7cb-74df8367974d\" (UID: \"2b699042-b83a-46b8-a7cb-74df8367974d\") " Nov 02 22:39:46.500443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.500420 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume" pod "2b699042-b83a-46b8-a7cb-74df8367974d" (UID: "2b699042-b83a-46b8-a7cb-74df8367974d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:46.502109 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.502082 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv" pod "2b699042-b83a-46b8-a7cb-74df8367974d" (UID: "2b699042-b83a-46b8-a7cb-74df8367974d"). InnerVolumeSpecName "kube-api-access-5bwxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:46.578749 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.578718 4285 generic.go:356] "Generic (PLEG): container finished" podID="2b699042-b83a-46b8-a7cb-74df8367974d" containerID="67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90" exitCode=0 Nov 02 22:39:46.578909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.578748 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" event={"ID":"2b699042-b83a-46b8-a7cb-74df8367974d","Type":"ContainerDied","Data":"67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90"} Nov 02 22:39:46.578909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.578770 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" Nov 02 22:39:46.578909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.578838 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh" event={"ID":"2b699042-b83a-46b8-a7cb-74df8367974d","Type":"ContainerDied","Data":"b4f10be5db673e873e503f954c62958e20b66fb4261bd0cc8a0a5d4fa6e9a331"} Nov 02 22:39:46.578909 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.578867 4285 scope.go:122] "RemoveContainer" containerID="67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90" Nov 02 22:39:46.580375 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.580354 4285 generic.go:356] "Generic (PLEG): container finished" podID="92648963-464b-4392-b018-f7769cd7e537" containerID="64438430f2113ac8eec7942d6a51b02e8789758b51bcd731b7d861d6335a7e90" exitCode=0 Nov 02 22:39:46.580463 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.580411 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" event={"ID":"92648963-464b-4392-b018-f7769cd7e537","Type":"ContainerDied","Data":"64438430f2113ac8eec7942d6a51b02e8789758b51bcd731b7d861d6335a7e90"} Nov 02 22:39:46.581497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.581482 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:39:46.588785 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.588770 4285 scope.go:122] "RemoveContainer" containerID="67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90" Nov 02 22:39:46.588994 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:39:46.588977 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90\": container with ID starting with 67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90 not found: ID does not exist" containerID="67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90" Nov 02 22:39:46.589033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.589006 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90"} err="failed to get container status \"67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90\": rpc error: code = NotFound desc = could not find container \"67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90\": container with ID starting with 67ca0338713410f0fb5aef62cd893bdc53b9abd9a0c1cac09456cf8bb7f05b90 not found: ID does not exist" Nov 02 22:39:46.601265 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.601242 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq\" (UniqueName: \"kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq\") pod \"92648963-464b-4392-b018-f7769cd7e537\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " Nov 02 22:39:46.601369 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.601358 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume\") pod \"92648963-464b-4392-b018-f7769cd7e537\" (UID: \"92648963-464b-4392-b018-f7769cd7e537\") " Nov 02 22:39:46.601558 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.601547 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b699042-b83a-46b8-a7cb-74df8367974d-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:46.601595 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.601561 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwxv\" (UniqueName: \"kubernetes.io/projected/2b699042-b83a-46b8-a7cb-74df8367974d-kube-api-access-5bwxv\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:46.601685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.601662 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume" pod "92648963-464b-4392-b018-f7769cd7e537" (UID: "92648963-464b-4392-b018-f7769cd7e537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:46.603230 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.603206 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq" pod "92648963-464b-4392-b018-f7769cd7e537" (UID: "92648963-464b-4392-b018-f7769cd7e537"). InnerVolumeSpecName "kube-api-access-hkmcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:46.671857 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.671827 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h"] Nov 02 22:39:46.672018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.672001 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" podUID="1981d04e-0e92-4985-ab50-ae0046c65ec3" containerName="maas-api" containerID="cri-o://3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10" gracePeriod=30 Nov 02 22:39:46.702035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.702011 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmcq\" (UniqueName: \"kubernetes.io/projected/92648963-464b-4392-b018-f7769cd7e537-kube-api-access-hkmcq\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:46.702035 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.702036 4285 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92648963-464b-4392-b018-f7769cd7e537-plugins-config-volume\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:46.796728 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.796694 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:46.817047 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.817002 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh"] Nov 02 22:39:46.819315 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.819288 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce65d173-747b4dfb7-5flkh"] Nov 02 22:39:46.920080 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:46.920057 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:39:47.003899 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.003869 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca\") pod \"1981d04e-0e92-4985-ab50-ae0046c65ec3\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " Nov 02 22:39:47.004062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.003987 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs\") pod \"1981d04e-0e92-4985-ab50-ae0046c65ec3\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " Nov 02 22:39:47.004062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.004028 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c\" (UniqueName: \"kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c\") pod \"1981d04e-0e92-4985-ab50-ae0046c65ec3\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " Nov 02 22:39:47.004062 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.004048 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\") pod \"1981d04e-0e92-4985-ab50-ae0046c65ec3\" (UID: \"1981d04e-0e92-4985-ab50-ae0046c65ec3\") " Nov 02 22:39:47.004273 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.004202 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca" pod "1981d04e-0e92-4985-ab50-ae0046c65ec3" (UID: "1981d04e-0e92-4985-ab50-ae0046c65ec3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:47.004542 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.004335 4285 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1981d04e-0e92-4985-ab50-ae0046c65ec3-openshift-service-ca\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:47.005947 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.005924 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs" pod "1981d04e-0e92-4985-ab50-ae0046c65ec3" (UID: "1981d04e-0e92-4985-ab50-ae0046c65ec3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:39:47.006023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.005982 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c" pod "1981d04e-0e92-4985-ab50-ae0046c65ec3" (UID: "1981d04e-0e92-4985-ab50-ae0046c65ec3"). InnerVolumeSpecName "kube-api-access-q8s9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:47.006023 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.006007 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls" pod "1981d04e-0e92-4985-ab50-ae0046c65ec3" (UID: "1981d04e-0e92-4985-ab50-ae0046c65ec3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:39:47.105471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.105437 4285 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-metrics-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:47.105471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.105465 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8s9c\" (UniqueName: \"kubernetes.io/projected/1981d04e-0e92-4985-ab50-ae0046c65ec3-kube-api-access-q8s9c\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:47.105471 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.105476 4285 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1981d04e-0e92-4985-ab50-ae0046c65ec3-maas-api-tls\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:39:47.178627 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.178553 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b699042-b83a-46b8-a7cb-74df8367974d" path="/var/lib/kubelet/pods/2b699042-b83a-46b8-a7cb-74df8367974d/volumes" Nov 02 22:39:47.593109 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.593055 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" event={"ID":"92648963-464b-4392-b018-f7769cd7e537","Type":"ContainerDied","Data":"ebb929d58ca456435737cd3b36c8c5594a7bdc1c5130a6255511d3203ea00e9b"} Nov 02 22:39:47.593343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.593061 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt" Nov 02 22:39:47.593343 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.593155 4285 scope.go:122] "RemoveContainer" containerID="64438430f2113ac8eec7942d6a51b02e8789758b51bcd731b7d861d6335a7e90" Nov 02 22:39:47.594988 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.594963 4285 generic.go:356] "Generic (PLEG): container finished" podID="1981d04e-0e92-4985-ab50-ae0046c65ec3" containerID="3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10" exitCode=0 Nov 02 22:39:47.595147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.595000 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" event={"ID":"1981d04e-0e92-4985-ab50-ae0046c65ec3","Type":"ContainerDied","Data":"3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10"} Nov 02 22:39:47.595147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.595034 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" event={"ID":"1981d04e-0e92-4985-ab50-ae0046c65ec3","Type":"ContainerDied","Data":"f7e76b2728f26b38b1b86e5ff98a77d03b9d67adabab4f4e50a8c787f54fd6b8"} Nov 02 22:39:47.595147 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.595004 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h" Nov 02 22:39:47.605572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.605547 4285 scope.go:122] "RemoveContainer" containerID="3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10" Nov 02 22:39:47.617490 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.617470 4285 scope.go:122] "RemoveContainer" containerID="3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10" Nov 02 22:39:47.617764 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:39:47.617737 4285 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10\": container with ID starting with 3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10 not found: ID does not exist" containerID="3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10" Nov 02 22:39:47.617832 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.617773 4285 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10"} err="failed to get container status \"3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10\": rpc error: code = NotFound desc = could not find container \"3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10\": container with ID starting with 3f84f02028548e0e73bcf0e5eeefad1f52484fd66707fa22a09caabb80f60e10 not found: ID does not exist" Nov 02 22:39:47.756292 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.756268 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h"] Nov 02 22:39:47.805575 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.805553 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce65d173-65556f5fc-s4g7h"] Nov 02 22:39:47.870320 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.870256 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt"] Nov 02 22:39:47.875299 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:47.875270 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce65d173-584546fb99-724lt"] Nov 02 22:39:48.285810 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:48.285783 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:48.575563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:48.575487 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:49.173825 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:49.173788 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1981d04e-0e92-4985-ab50-ae0046c65ec3" path="/var/lib/kubelet/pods/1981d04e-0e92-4985-ab50-ae0046c65ec3/volumes" Nov 02 22:39:49.174330 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:49.174312 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92648963-464b-4392-b018-f7769cd7e537" path="/var/lib/kubelet/pods/92648963-464b-4392-b018-f7769cd7e537/volumes" Nov 02 22:39:50.785933 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:50.785897 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:51.904018 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:51.903984 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:52.348577 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348549 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd"] Nov 02 22:39:52.348943 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348932 4285 cpu_manager.go:414] "containerMap: removing container" podUID="2b699042-b83a-46b8-a7cb-74df8367974d" containerName="payload-processing" Nov 02 22:39:52.348995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348945 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b699042-b83a-46b8-a7cb-74df8367974d" containerName="payload-processing" Nov 02 22:39:52.348995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348959 4285 cpu_manager.go:414] "containerMap: removing container" podUID="1981d04e-0e92-4985-ab50-ae0046c65ec3" containerName="maas-api" Nov 02 22:39:52.348995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348964 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1981d04e-0e92-4985-ab50-ae0046c65ec3" containerName="maas-api" Nov 02 22:39:52.348995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348980 4285 cpu_manager.go:414] "containerMap: removing container" podUID="92648963-464b-4392-b018-f7769cd7e537" containerName="payload-pre-processing" Nov 02 22:39:52.348995 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.348985 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92648963-464b-4392-b018-f7769cd7e537" containerName="payload-pre-processing" Nov 02 22:39:52.349194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.349047 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="92648963-464b-4392-b018-f7769cd7e537" containerName="payload-pre-processing" Nov 02 22:39:52.349194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.349060 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b699042-b83a-46b8-a7cb-74df8367974d" containerName="payload-processing" Nov 02 22:39:52.349194 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.349067 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="1981d04e-0e92-4985-ab50-ae0046c65ec3" containerName="maas-api" Nov 02 22:39:52.353459 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.353440 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.406868 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.406836 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-0f09fe-openshift-default-dockercfg-k7wdc\"" Nov 02 22:39:52.460694 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.460663 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.460891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.460707 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.460891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.460807 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.460891 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.460880 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.461022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.460968 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.461022 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.461012 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.461121 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.461038 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.461180 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.461152 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.461219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.461181 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsr4\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.482117 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.482074 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd"] Nov 02 22:39:52.528815 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.528781 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:52.562178 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562143 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562202 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562249 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562283 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562335 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562316 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562341 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsr4\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562377 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562407 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562457 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562563 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562541 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562642 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562823 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562796 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562890 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562818 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.562983 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.562967 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.564615 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.564594 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.564804 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.564786 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.619009 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.618952 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.619163 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.619141 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsr4\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4\") pod \"e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.676205 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.676169 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:52.844044 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:52.844013 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:53.096183 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.096159 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd"] Nov 02 22:39:53.098079 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:39:53.098053 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b57ed3_fc09_454b_97ff_6978fbce0c43.slice/crio-1a77b7e3eff92116f3b5db571722db18d4ed553044f3dad23f10bd293a557a2a WatchSource:0}: Error finding container 1a77b7e3eff92116f3b5db571722db18d4ed553044f3dad23f10bd293a557a2a: Status 404 returned error can't find the container with id 1a77b7e3eff92116f3b5db571722db18d4ed553044f3dad23f10bd293a557a2a Nov 02 22:39:53.100175 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.100116 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:39:53.100268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.100206 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:39:53.100268 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.100235 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:39:53.639204 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.639168 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" event={"ID":"b5b57ed3-fc09-454b-97ff-6978fbce0c43","Type":"ContainerStarted","Data":"c19ad38367703f951237547aff60b19338e4514169a6253a3fed79c7e6e1dc4b"} Nov 02 22:39:53.639378 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.639219 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" event={"ID":"b5b57ed3-fc09-454b-97ff-6978fbce0c43","Type":"ContainerStarted","Data":"1a77b7e3eff92116f3b5db571722db18d4ed553044f3dad23f10bd293a557a2a"} Nov 02 22:39:53.676404 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.676378 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:53.755410 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:53.755357 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podStartSLOduration=1.755344469 podStartE2EDuration="1.755344469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:53.75443715 +0000 UTC m=+3932.715987469" watchObservedRunningTime="2026-11-02 22:39:53.755344469 +0000 UTC m=+3932.716894787" Nov 02 22:39:54.682150 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:54.682103 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:55.649497 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:55.649468 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:55.650238 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:55.650220 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:39:56.491835 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:56.491804 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:56.661098 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:56.661068 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:56.837680 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:56.837576 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:59.720799 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:59.720763 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:59.901781 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:39:59.901746 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:03.275861 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:03.275822 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:03.438531 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:03.438501 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:03.722520 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:03.722492 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:08.152110 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:08.152077 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:12.965826 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:12.965787 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:16.006659 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:40:16.006617 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:40:17.650720 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:17.648835 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:20.784507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:20.784477 4285 generic.go:356] "Generic (PLEG): container finished" podID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerID="c19ad38367703f951237547aff60b19338e4514169a6253a3fed79c7e6e1dc4b" exitCode=0 Nov 02 22:40:20.785042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:20.784556 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" event={"ID":"b5b57ed3-fc09-454b-97ff-6978fbce0c43","Type":"ContainerDied","Data":"c19ad38367703f951237547aff60b19338e4514169a6253a3fed79c7e6e1dc4b"} Nov 02 22:40:20.785042 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:20.784977 4285 scope.go:122] "RemoveContainer" containerID="c19ad38367703f951237547aff60b19338e4514169a6253a3fed79c7e6e1dc4b" Nov 02 22:40:20.786144 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:20.786101 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:40:20.786219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:20.786179 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:40:20.786219 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:20.786212 4285 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:40:21.261686 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:21.261643 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:21.790898 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:21.790864 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" event={"ID":"b5b57ed3-fc09-454b-97ff-6978fbce0c43","Type":"ContainerStarted","Data":"960470a8166bac130a1ad43f0b74272b833348ca2ad80e8e1b89e0a75c8712dc"} Nov 02 22:40:22.677199 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:22.677166 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:40:22.678452 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:22.678425 4285 patch_prober.go:29] interesting pod/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:40:22.678572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:22.678486 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 22:40:22.729146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:22.728469 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:23.677332 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:23.677294 4285 patch_prober.go:29] interesting pod/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:40:23.677734 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:23.677349 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 22:40:24.433221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:24.433175 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:24.677507 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:24.677473 4285 patch_prober.go:29] interesting pod/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:40:24.677981 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:24.677532 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 22:40:24.838024 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:24.837984 4285 scope.go:122] "RemoveContainer" containerID="16cc6783eb15da27f499982c029e8ce369d743ec3646d464d319adb0c3e14a13" Nov 02 22:40:24.846242 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:24.846216 4285 scope.go:122] "RemoveContainer" containerID="c8c6534b1b24d3ca96d0d5adc70c719a6a99c345824967ec21ce14c5b7f94109" Nov 02 22:40:24.853806 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:24.853789 4285 scope.go:122] "RemoveContainer" containerID="d98f31bfea73ec85d7153f6b363e57c16053de8a2a630f6d6c61de53e49e3c27" Nov 02 22:40:25.649512 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:25.649481 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:40:25.676625 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:25.676593 4285 patch_prober.go:29] interesting pod/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:40:25.676760 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:25.676639 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 22:40:26.676801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:26.676766 4285 patch_prober.go:29] interesting pod/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 02 22:40:26.677245 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:26.676822 4285 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 02 22:40:27.314587 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:27.314536 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:27.680770 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:27.680700 4285 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:40:27.681744 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:27.681723 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:40:32.111915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:32.111882 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:35.046684 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:35.046653 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7"] Nov 02 22:40:35.096085 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:35.096056 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf2a4ca-g45v7"] Nov 02 22:40:35.172070 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:35.172040 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9caa702-5ca2-4614-88d9-1506d6858ea4" path="/var/lib/kubelet/pods/d9caa702-5ca2-4614-88d9-1506d6858ea4/volumes" Nov 02 22:40:36.827693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:36.827659 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:41.618193 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:41.618163 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:46.003594 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:40:46.003558 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:40:46.457634 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:46.457604 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:48.385552 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:48.385517 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:51.255853 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:51.255797 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:55.941612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:40:55.941572 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:00.733768 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:00.733733 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:05.532221 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:05.532092 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:10.458523 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:10.458487 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:15.235046 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:15.235001 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:16.005001 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:41:16.004967 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:41:20.025342 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:20.025308 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:24.710732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:24.710685 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:24.871457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:24.871419 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:29.548769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:29.548672 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:34.308697 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:34.308658 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:36.271971 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:36.271923 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:39.121782 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:39.121751 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:43.838501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:43.838465 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:46.004506 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:41:46.004467 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:41:49.054399 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:49.054366 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:53.835637 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:53.835599 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:56.107986 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:56.107950 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:56.261333 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:41:56.261294 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:03.095496 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:03.095461 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:04.595270 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:04.595236 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:04.982435 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:04.982408 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:05.609666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:05.609629 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:11.738186 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:11.738153 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:11.912680 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:11.912647 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:12.030801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:12.030726 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd"] Nov 02 22:42:12.031029 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:12.031006 4285 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" containerID="cri-o://960470a8166bac130a1ad43f0b74272b833348ca2ad80e8e1b89e0a75c8712dc" gracePeriod=30 Nov 02 22:42:12.261923 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:12.261890 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:12.683097 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:12.683067 4285 patch_prober.go:29] interesting pod/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:42:12.683263 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:12.683147 4285 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:42:16.006235 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:42:16.006200 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:42:17.408952 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.408915 4285 generic.go:356] "Generic (PLEG): container finished" podID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerID="960470a8166bac130a1ad43f0b74272b833348ca2ad80e8e1b89e0a75c8712dc" exitCode=0 Nov 02 22:42:17.409607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.409161 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" event={"ID":"b5b57ed3-fc09-454b-97ff-6978fbce0c43","Type":"ContainerDied","Data":"960470a8166bac130a1ad43f0b74272b833348ca2ad80e8e1b89e0a75c8712dc"} Nov 02 22:42:17.409607 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.409240 4285 scope.go:122] "RemoveContainer" containerID="c19ad38367703f951237547aff60b19338e4514169a6253a3fed79c7e6e1dc4b" Nov 02 22:42:17.537801 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.537085 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:17.544145 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.543099 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698588 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698655 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698734 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698809 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698847 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698890 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698926 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.698959 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.699001 4285 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data\") pod \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\" (UID: \"b5b57ed3-fc09-454b-97ff-6978fbce0c43\") " Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.699913 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.700144 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:42:17.700769 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.700417 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:17.701612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.700924 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:42:17.701612 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.700938 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:42:17.702624 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.702595 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:42:17.702940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.702916 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "kube-api-access-wcsr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:17.702940 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.702934 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:17.703202 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.703184 4285 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy" pod "b5b57ed3-fc09-454b-97ff-6978fbce0c43" (UID: "b5b57ed3-fc09-454b-97ff-6978fbce0c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:42:17.799809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799777 4285 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-token\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.799809 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799805 4285 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-credential-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799819 4285 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-socket\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799828 4285 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-podinfo\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799838 4285 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-workload-certs\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799846 4285 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istiod-ca-cert\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799855 4285 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsr4\" (UniqueName: \"kubernetes.io/projected/b5b57ed3-fc09-454b-97ff-6978fbce0c43-kube-api-access-wcsr4\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799864 4285 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-envoy\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:17.800033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:17.799872 4285 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b57ed3-fc09-454b-97ff-6978fbce0c43-istio-data\") on node \"ip-10-0-131-47.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 22:42:18.486148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:18.485943 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" event={"ID":"b5b57ed3-fc09-454b-97ff-6978fbce0c43","Type":"ContainerDied","Data":"1a77b7e3eff92116f3b5db571722db18d4ed553044f3dad23f10bd293a557a2a"} Nov 02 22:42:18.486148 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:18.486068 4285 scope.go:122] "RemoveContainer" containerID="960470a8166bac130a1ad43f0b74272b833348ca2ad80e8e1b89e0a75c8712dc" Nov 02 22:42:18.486732 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:18.486302 4285 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd" Nov 02 22:42:18.714840 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:18.714784 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd"] Nov 02 22:42:18.719222 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:18.719194 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0f09fe-openshift-default-578cccfbdf-wtnsd"] Nov 02 22:42:19.172553 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:19.172520 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" path="/var/lib/kubelet/pods/b5b57ed3-fc09-454b-97ff-6978fbce0c43/volumes" Nov 02 22:42:24.899633 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:24.899604 4285 scope.go:122] "RemoveContainer" containerID="b42cb0c2be56631b1eb106b374d23a94b1a12aa1bc2649dd0d53c7cd42ce57a4" Nov 02 22:42:24.907652 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:24.907631 4285 scope.go:122] "RemoveContainer" containerID="c6a45c9c105662571a441df0bc2f8711acd6738be0885a6d21f768c68401b9ad" Nov 02 22:42:25.096764 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:25.096735 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:28.339051 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:28.339016 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:35.052693 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:35.052595 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd"] Nov 02 22:42:35.100234 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:35.100198 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-2ztpd"] Nov 02 22:42:35.173706 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:42:35.173663 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd63805-8355-4807-89da-f9e8b6f4cba1" path="/var/lib/kubelet/pods/dbd63805-8355-4807-89da-f9e8b6f4cba1/volumes" Nov 02 22:42:46.004991 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:42:46.004944 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:43:03.088666 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:03.088636 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5"] Nov 02 22:43:03.136715 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:03.136690 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-czxq5"] Nov 02 22:43:03.171574 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:03.171548 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595aec09-dc34-49b8-834f-1821f988548f" path="/var/lib/kubelet/pods/595aec09-dc34-49b8-834f-1821f988548f/volumes" Nov 02 22:43:16.004107 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:43:16.004073 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:43:25.429168 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:25.429117 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:33.050570 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:33.050534 4285 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df"] Nov 02 22:43:33.099639 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:33.099609 4285 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852dca62-fk9df"] Nov 02 22:43:33.172665 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:33.172630 4285 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d3e40a-8ac7-42ec-933c-537d1673bdfe" path="/var/lib/kubelet/pods/38d3e40a-8ac7-42ec-933c-537d1673bdfe/volumes" Nov 02 22:43:36.765562 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:36.765529 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-648dd6fb45-v9bkq_5c47b892-df35-4707-9e8e-45c0999acf87/manager/0.log" Nov 02 22:43:38.867915 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:38.867890 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pbvwm_cc7a35e2-d3a3-4f3a-a278-84423b5ea520/server/0.log" Nov 02 22:43:40.431465 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:40.431436 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b8648dd4-jfblg_e4a6cacc-a52e-4add-8189-67f200ca638a/maas-api/0.log" Nov 02 22:43:41.387955 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:41.387923 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hcdfl_1f756716-a3b4-44e3-a23c-53860c6385d9/postgres/0.log" Nov 02 22:43:41.619233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:41.619202 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:41.775301 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:41.775271 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:44.101631 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:44.101602 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kdxrm_3a657361-0cf8-460f-9f1c-2cb6dc0cc194/manager/0.log" Nov 02 22:43:45.903935 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:45.903900 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q96hh_c6611b39-cd2a-4a7a-a199-d198f1a94049/discovery/0.log" Nov 02 22:43:46.004112 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:43:46.004080 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:43:46.174527 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:46.174443 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d64f9d85b-crltx_4e8d0ba1-1b77-4850-b6ad-960a92533271/kube-auth-proxy/0.log" Nov 02 22:43:46.614368 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:46.614335 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-blcrl_6eb34991-dfd5-4e0f-9639-0aad1f6f4447/istio-proxy/0.log" Nov 02 22:43:47.464526 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:47.464497 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-mcx5l_cd208f3f-912d-4600-982d-22c0ff27389e/istio-proxy/0.log" Nov 02 22:43:47.731504 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:47.731389 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-sf2hx_dc0ddbc9-69dd-411e-bf0f-423369dc5b2d/istio-proxy/0.log" Nov 02 22:43:49.399134 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:49.399078 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rdz8f_34cdf2d8-2345-46fe-b006-fb6f05a86bf2/payload-pre-processing/0.log" Nov 02 22:43:49.599312 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:49.599285 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9l4v9_6730dff9-7d15-4a6a-bb30-63bccf531f4c/payload-processing/0.log" Nov 02 22:43:50.626034 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:50.625992 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6_cc6364ff-f21b-462b-a93e-f71e85c3e0f8/main/0.log" Nov 02 22:43:51.094142 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:51.094107 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tw887_1f8b5643-8dd6-45ee-ba29-6d6d573eaad6/main/0.log" Nov 02 22:43:51.552548 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:51.552523 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp_615e6adb-9482-405a-85f3-98d3af5df4cd/main/0.log" Nov 02 22:43:52.025373 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:43:52.025332 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x_6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41/main/0.log" Nov 02 22:44:01.885998 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:01.885964 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-648dd6fb45-v9bkq_5c47b892-df35-4707-9e8e-45c0999acf87/manager/0.log" Nov 02 22:44:03.565851 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:03.565755 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pbvwm_cc7a35e2-d3a3-4f3a-a278-84423b5ea520/server/0.log" Nov 02 22:44:04.822139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:04.822097 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b8648dd4-jfblg_e4a6cacc-a52e-4add-8189-67f200ca638a/maas-api/0.log" Nov 02 22:44:05.764685 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:05.764649 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hcdfl_1f756716-a3b4-44e3-a23c-53860c6385d9/postgres/0.log" Nov 02 22:44:07.931050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:07.931015 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kdxrm_3a657361-0cf8-460f-9f1c-2cb6dc0cc194/manager/0.log" Nov 02 22:44:09.474736 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:09.474707 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q96hh_c6611b39-cd2a-4a7a-a199-d198f1a94049/discovery/0.log" Nov 02 22:44:09.743139 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:09.743043 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d64f9d85b-crltx_4e8d0ba1-1b77-4850-b6ad-960a92533271/kube-auth-proxy/0.log" Nov 02 22:44:10.184384 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:10.184278 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-blcrl_6eb34991-dfd5-4e0f-9639-0aad1f6f4447/istio-proxy/0.log" Nov 02 22:44:11.045425 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:11.045397 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-mcx5l_cd208f3f-912d-4600-982d-22c0ff27389e/istio-proxy/0.log" Nov 02 22:44:11.313059 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:11.312977 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-sf2hx_dc0ddbc9-69dd-411e-bf0f-423369dc5b2d/istio-proxy/0.log" Nov 02 22:44:13.031673 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:13.031627 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rdz8f_34cdf2d8-2345-46fe-b006-fb6f05a86bf2/payload-pre-processing/0.log" Nov 02 22:44:13.249233 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:13.249185 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9l4v9_6730dff9-7d15-4a6a-bb30-63bccf531f4c/payload-processing/0.log" Nov 02 22:44:14.144443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:14.144415 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l8dh6_cc6364ff-f21b-462b-a93e-f71e85c3e0f8/main/0.log" Nov 02 22:44:14.676838 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:14.676807 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tw887_1f8b5643-8dd6-45ee-ba29-6d6d573eaad6/main/0.log" Nov 02 22:44:15.179894 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:15.179861 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89gccp_615e6adb-9482-405a-85f3-98d3af5df4cd/main/0.log" Nov 02 22:44:15.655457 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:15.655429 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zdh9x_6f8ab551-6d4a-4ad1-a3c5-c662ed0adc41/main/0.log" Nov 02 22:44:15.907967 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:15.907869 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:16.004615 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:44:16.004583 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:44:24.947043 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:24.947008 4285 scope.go:122] "RemoveContainer" containerID="ec64cdf3b6b9c17ddee261c0136bff67fbb3bf91bad552675cc407e08fed7a0d" Nov 02 22:44:24.955248 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:24.955225 4285 scope.go:122] "RemoveContainer" containerID="0ad9cbe0068ca067e3f1e8f59b1c224057db031b1c53292a9e445767018dc5f4" Nov 02 22:44:25.750957 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:25.750918 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:46.004577 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:44:46.004540 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:44:47.491946 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:47.491918 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9l4v9_6730dff9-7d15-4a6a-bb30-63bccf531f4c/payload-processing/0.log" Nov 02 22:44:47.711057 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:44:47.711031 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9l4v9_6730dff9-7d15-4a6a-bb30-63bccf531f4c/payload-processing/0.log" Nov 02 22:45:16.004196 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:45:16.004160 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:45:24.977476 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:24.977441 4285 scope.go:122] "RemoveContainer" containerID="ef43ca3c56dc1e5597bb5eb73125e43ec483f9193ec8e9c8f088d93a0a1fb03c" Nov 02 22:45:25.310397 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:25.310325 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9l4v9_6730dff9-7d15-4a6a-bb30-63bccf531f4c/payload-processing/0.log" Nov 02 22:45:25.529833 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:25.529804 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9l4v9_6730dff9-7d15-4a6a-bb30-63bccf531f4c/payload-processing/0.log" Nov 02 22:45:25.920160 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:25.920111 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:27.511308 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:27.511275 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:35.472916 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:35.472846 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lmrxw_befb271e-da36-470a-ae6e-b7ce5dabf394/global-pull-secret-syncer/0.log" Nov 02 22:45:35.538373 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:35.538348 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:35.784374 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:35.784299 4285 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:37.960226 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:37.960192 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rtnjt_2615d298-b340-465b-b9b0-f66aa0f59628/konnectivity-agent/0.log" Nov 02 22:45:39.798522 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:39.798496 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-47.us-west-1.compute.internal_ef8aa625af233e1fa89cb16cb4aca78e/haproxy/0.log" Nov 02 22:45:44.861953 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.861910 4285 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg"] Nov 02 22:45:44.862341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.862328 4285 cpu_manager.go:414] "containerMap: removing container" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" Nov 02 22:45:44.862341 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.862340 4285 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" Nov 02 22:45:44.862443 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.862432 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" Nov 02 22:45:44.862481 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.862447 4285 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b57ed3-fc09-454b-97ff-6978fbce0c43" containerName="istio-proxy" Nov 02 22:45:44.865390 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.865376 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.879897 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.879876 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-proc\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.880004 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.879905 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-lib-modules\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.880044 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.880018 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-sys\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.880077 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.880051 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-podres\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.880146 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.880120 4285 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlchw\" (UniqueName: \"kubernetes.io/projected/6f71766c-1590-4c4e-bbb8-154e10ebf33b-kube-api-access-mlchw\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.917297 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.917269 4285 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lxt6h\"/\"default-dockercfg-q6zgg\"" Nov 02 22:45:44.967008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.966982 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lxt6h\"/\"openshift-service-ca.crt\"" Nov 02 22:45:44.967008 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.966982 4285 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lxt6h\"/\"kube-root-ca.crt\"" Nov 02 22:45:44.969033 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.969011 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg"] Nov 02 22:45:44.981161 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981112 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlchw\" (UniqueName: \"kubernetes.io/projected/6f71766c-1590-4c4e-bbb8-154e10ebf33b-kube-api-access-mlchw\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981193 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-proc\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981219 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-lib-modules\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981286 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981270 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-sys\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981288 4285 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-podres\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981306 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-proc\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981386 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-podres\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981395 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-lib-modules\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:44.981468 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:44.981389 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f71766c-1590-4c4e-bbb8-154e10ebf33b-sys\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:45.037334 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:45.037308 4285 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlchw\" (UniqueName: \"kubernetes.io/projected/6f71766c-1590-4c4e-bbb8-154e10ebf33b-kube-api-access-mlchw\") pod \"perf-node-gather-daemonset-8dhcg\" (UID: \"6f71766c-1590-4c4e-bbb8-154e10ebf33b\") " pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:45.177449 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:45.177373 4285 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:45.544276 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:45.544252 4285 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg"] Nov 02 22:45:45.546648 ip-10-0-131-47 kubenswrapper[4285]: W1102 22:45:45.546621 4285 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f71766c_1590_4c4e_bbb8_154e10ebf33b.slice/crio-b6b631c935cd945698ffa6c8db40d2c849504000c0b165e203ed82e4bab96007 WatchSource:0}: Error finding container b6b631c935cd945698ffa6c8db40d2c849504000c0b165e203ed82e4bab96007: Status 404 returned error can't find the container with id b6b631c935cd945698ffa6c8db40d2c849504000c0b165e203ed82e4bab96007 Nov 02 22:45:45.984884 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:45.984855 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kdxrm_3a657361-0cf8-460f-9f1c-2cb6dc0cc194/manager/0.log" Nov 02 22:45:46.004957 ip-10-0-131-47 kubenswrapper[4285]: E1102 22:45:46.004924 4285 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-276px" containerName="extractor" Nov 02 22:45:46.476516 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:46.476484 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" event={"ID":"6f71766c-1590-4c4e-bbb8-154e10ebf33b","Type":"ContainerStarted","Data":"10348a9742ad1ea0a7aae621c104f8a3e16bc6809ca9c5192d2255eeda4d7a55"} Nov 02 22:45:46.476711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:46.476541 4285 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" event={"ID":"6f71766c-1590-4c4e-bbb8-154e10ebf33b","Type":"ContainerStarted","Data":"b6b631c935cd945698ffa6c8db40d2c849504000c0b165e203ed82e4bab96007"} Nov 02 22:45:46.476711 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:46.476684 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:46.591050 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:46.591007 4285 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" podStartSLOduration=2.590994559 podStartE2EDuration="2.590994559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:46.589179275 +0000 UTC m=+4285.550729594" watchObservedRunningTime="2026-11-02 22:45:46.590994559 +0000 UTC m=+4285.552544879" Nov 02 22:45:49.607701 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:49.607673 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/alertmanager/0.log" Nov 02 22:45:50.021014 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:50.020988 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/config-reloader/0.log" Nov 02 22:45:50.433604 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:50.433537 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/kube-rbac-proxy-web/0.log" Nov 02 22:45:50.845831 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:50.845806 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/kube-rbac-proxy/0.log" Nov 02 22:45:51.258413 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:51.258390 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/kube-rbac-proxy-metric/0.log" Nov 02 22:45:51.671006 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:51.670937 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/prom-label-proxy/0.log" Nov 02 22:45:52.083572 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:52.083546 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3c02fc47-54b8-4f5a-855b-d71972658fc5/init-config-reloader/0.log" Nov 02 22:45:52.490501 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:52.490475 4285 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lxt6h/perf-node-gather-daemonset-8dhcg" Nov 02 22:45:54.210747 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:54.210723 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dc98b96fb-mslq5_8b619304-95b7-43ce-b2f2-5b7d6e37ed59/metrics-server/0.log" Nov 02 22:45:57.542707 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:57.542679 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrc7x_2e2fee67-70fc-4aec-9357-cce7e8461ef3/node-exporter/0.log" Nov 02 22:45:57.957420 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:57.957392 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrc7x_2e2fee67-70fc-4aec-9357-cce7e8461ef3/kube-rbac-proxy/0.log" Nov 02 22:45:58.370493 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:45:58.370421 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xrc7x_2e2fee67-70fc-4aec-9357-cce7e8461ef3/init-textfile/0.log" Nov 02 22:46:02.844550 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:02.844517 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vv9jh_54ee2947-1446-4a7f-a3e6-af59743823e3/prometheus-operator/0.log" Nov 02 22:46:03.253733 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:03.253700 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vv9jh_54ee2947-1446-4a7f-a3e6-af59743823e3/kube-rbac-proxy/0.log" Nov 02 22:46:03.668432 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:03.668354 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hwpp8_9e4ccef0-1600-4f94-883b-af10ec8699e7/prometheus-operator-admission-webhook/0.log" Nov 02 22:46:04.086519 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:04.086492 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b48b8bb5-7j56s_a96d8019-dc97-4a8e-9fca-be90d7795bcb/telemeter-client/0.log" Nov 02 22:46:04.499577 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:04.499550 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b48b8bb5-7j56s_a96d8019-dc97-4a8e-9fca-be90d7795bcb/reload/0.log" Nov 02 22:46:04.912655 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:04.912586 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86b48b8bb5-7j56s_a96d8019-dc97-4a8e-9fca-be90d7795bcb/kube-rbac-proxy/0.log" Nov 02 22:46:07.752303 ip-10-0-131-47 kubenswrapper[4285]: I1102 22:46:07.752259 4285 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-q96hh_c6611b39-cd2a-4a7a-a199-d198f1a94049/discovery/0.log"